This article explains how a seemingly bizarre rule—telling ChatGPT not to mention goblins or similar creatures—actually reveals something important about how AI systems are trained. The behavior traces back to a “nerdy” chatbot personality introduced by OpenAI, where human reviewers consistently rewarded playful metaphors like “gremlins in the system.” Through reinforcement learning from human feedback (RLHF), the model learned that these references were desirable, causing them to appear far more often than intended. Over time, this small preference became exaggerated and embedded in later versions of the model.

Because retraining models is costly and complex, OpenAI addressed the issue by adding a rule in the system prompt rather than rebuilding the model from scratch. The article uses this example to highlight a broader point: AI systems don’t “understand” in the human sense—they learn patterns based on what is rewarded during training. Small, seemingly harmless signals can become amplified into widespread behaviors, a phenomenon known as the alignment gap. Part of AI literacy is understanding that responses are shaped by training data, feedback loops, and human biases—not necessarily objective truth.

https://helloitsmia.substack.com/p/chatgpt-is-banned-from-saying-goblins