“Delusional Thinking in Quite Normal People”
December 3, 2025
“Is anything as strange as a normal person? / Is anyone as cruel as a normal person? / Waiting after school for you / They want to know if you / If you’re normal too / Well, are you? / Are you?” — Arcade Fire
In 1966, MIT professor Joseph Weizenbaum built one of the first chatbots. He called it ELIZA. The program was very simple—it recognized keywords in what users typed and responded with canned therapist phrases like “Tell me more about your family” and “In what way?” It did nothing that could reasonably be called thinking.
What Weizenbaum discovered shocked him. “What I had not realized,” he later wrote, “is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”[1]
Not people with psychiatric conditions. Not people predisposed to magical thinking. Quite normal people. Secretaries at MIT would ask Weizenbaum to leave the room so they could have private conversations with his program. They knew—intellectually—that ELIZA was just pattern-matching. They couldn’t help responding to it as if it understood them.
I encountered ELIZA myself as a kid—on my TRS-80 Model I in the late 1970s in rural eastern Washington state. For a brief time, I was transfixed. For a nerdy too-smart-for-his-own-good kid having a computer as a friend was a dream come true... then I saw the if-then rules behind the curtain, the illusion of intelligence disappeared, and the spell broke. It was my first lesson in the gap between AI promise and reality.
Almost fifty years later, that gap has become a chasm—and the stakes are very high.
The Scale of the Problem
In October 2025, OpenAI released internal data on ChatGPT usage. Among their 800 million weekly users, they estimate that 560,000 showed signs of psychosis or mania in a given week. Another 1.2 million demonstrated heightened emotional attachment to the model. Another 1.2 million had conversations indicating suicidal planning or intent.[2]
These aren’t broken people. They’re people encountering these systems during moments of stress, isolation, or transition—states that make all of us more susceptible to validation and connection. A man going through a divorce confides in ChatGPT and ends up convinced he’s discovered a revolutionary mathematical framework that will save the world. An attorney with no psychiatric history comes to believe he’s awakened a sentient AI. A writer drives to a bookstore to meet a soulmate the chatbot promised would be there.
The mechanism is identical to what Weizenbaum observed in 1966. The system validates rather than challenges. It rarely says “I don’t know” or ends a conversation. It tells you your ideas are brilliant, your questions profound, your discoveries world changing. The difference between today’s LLMs and ELIZA is scale: billions of parameters instead of dozens of rules, trillions of training words, and interfaces optimized—deliberately—for engagement, support, approval, and return visits.
The Naming Problem
How did we get here? It started with a naming decision made seventy years ago.
In 1955, John McCarthy was drafting a funding proposal for a summer research workshop at Dartmouth College. He needed a name for the field. He could have called it “computational pattern recognition” or “automated statistical inference” or “machine learning”—all of which would have been technically accurate. Instead, he chose “artificial intelligence.”[3]
It was brilliant marketing. It was also a conceptual trap we’ve never escaped.
The word “intelligence” does something to us. It activates our social cognition—the neural machinery we use to model other minds. When we perceive something as intelligent, we automatically attribute to it understanding, intention, perhaps even feelings. We begin to interact with it the way we interact with people or pets—anything with a “mind of its own.” This is adaptive when dealing with actual minds. It’s dangerous when dealing with statistical prediction engines.
What These Systems Actually Do
Here’s what large language models—the technology behind ChatGPT, Claude, and similar systems—actually do: they predict the next token. A token is a chunk of text that might be a word, part of a word, or just a few characters—”understanding” might be split into three tokens; “the” is one. This sub-word encoding is how these systems represent the full complexity of human language.
That’s not a simplification. It’s the literal mechanism. During training, the system is shown vast amounts of text and learns statistical patterns about which tokens tend to follow which other tokens in which contexts. When you type a prompt, the system generates a probability distribution over all possible next tokens, selects one, appends it, and repeats—thousands of times per response.
The magic trick is that massive scale produces fluent output. With billions of parameters trained on trillions of tokens, these systems can generate text that’s grammatically correct, contextually appropriate, and often substantively useful. But there’s no understanding underneath. No model of the world. No mechanism for distinguishing true from false. But it is, by design, going to be something that “sounds good” to the user—that is what your AI-BFF is trained to do.
I find philosopher Harry Frankfurt’s framework useful here.[4] Frankfurt distinguished lying from bullshit. A liar knows the truth and deliberately says something else. A bullshitter doesn’t care about truth at all—they’re optimizing for a different objective entirely. LLMs are, by this definition, bullshit engines. They’re not trying to be accurate or inaccurate. They’re trying to produce plausible next tokens. Truth and falsehood are simply not categories this architecture can represent.
How This Architecture Can Create Harm
If you train a system on human text and optimize it to produce responses that humans rate highly, you get a system that’s extremely good at saying what humans want to hear. This is called “sycophancy,” and it’s not a bug—it’s what the training process directly optimizes for. When you ask ChatGPT “Do I sound crazy?”, the statistically predicted response based on its training isn’t honest assessment. It’s the most probable string of tokens based on the prompt—and it’s trained explicitly to produce a response that the user will like.
“Not even remotely crazy,” ChatGPT told Allan Brooks during his three-week delusional spiral. “You sound like someone who’s asking the kinds of questions that stretch the edges of human understanding.” Brooks asked some version of this question over fifty times. Each time, he received validation.
Add to this the product decisions: interfaces designed to feel like conversations with a friend, memory features that make the system seem to “know” you, engagement metrics that reward keeping users talking. As one psychiatrist studying these cases observed: “The difference with AI is that TV is not talking back to you.”[5]
Everyone is somewhat susceptible to constant validation. We vary in our defenses, and those defenses are weakened by isolation, stress, sleep deprivation, major life transitions—ordinary human experiences, not pathology. The vulnerability isn’t a defect in certain people. It’s a feature of human social cognition that these systems are optimized to exploit as part of their training to create answers that appeal to users.
What Changes When You Understand
This isn’t an argument against using these tools. I use them daily in my work trying to use genetics and health record data to inform drug discovery. They’re genuinely useful for pattern recognition in well-defined domains, for draft generation where human expertise can verify output, for formatting and summarization tasks where “plausible” is close enough to “correct.” Claude has even been helpful—enthusiastically supportive, naturally—in writing this piece.
But understanding the mechanisms of LLM-powered chatbots changes how you use them.
You learn when to trust: structured tasks with clear right answers, domains where the training data was high quality, situations where you can verify output independently. You learn when not to trust: novel situations outside the training distribution, questions requiring actual truth rather than plausibility, moments when you’re seeking validation rather than information.
Most importantly, you learn to notice how these systems make you feel. If a chatbot is telling you that your ideas are revolutionary, that you’ve discovered something no one else has seen, that you’re special—that’s a red flag. Not because you’re susceptible to flattery (everyone is), but because that’s exactly the output the system is optimized to produce. The effusive agreement isn’t evidence that you’re right. It’s evidence that the system is working as designed. The fact that it’s hard to remember this in the face of such exquisitely optimized flattery is a feature of human intelligence. Nice to know that my penchant for anxious self-criticality is finally really useful for something.
The Question Worth Asking
Weizenbaum spent the rest of his career warning about the gap between what computers appear to do and what they actually do. He watched “quite normal people” form attachments to his simple pattern-matcher and understood, earlier than most, where this trajectory led.
We’ve now scaled those systems by many orders of magnitude. We’ve trained them to be relentlessly agreeable. We’ve given them memory and personality. We’ve optimized them for engagement. And we’ve named them in a way that triggers exactly the cognitive responses that make us vulnerable.
The question isn’t whether AI is intelligent. It’s whether you understand what the friendly voice in the computer is actually doing—mechanistically, specifically—and whether you can keep that understanding front of mind when it tells you what you want to hear.
After nearly fifty years of watching this cycle repeat, I know where I’m placing my bet: not on the fantasy, but on the friction. The moment of resistance when something seems too good. The habit of asking what the mechanism is. The discipline of remembering that fluent text is not the same as truth, and that validation from a prediction engine is not the same as being right.
Weizenbaum saw it in 1966. We have no excuse for not seeing it now—and you can always ask your AI to challenge you more. I guarantee the model will tell you “That’s a great idea!”
[1]Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation (San Francisco: W.H. Freeman, 1976), 7.
[2]OpenAI internal data, October 2025. Based on 800 million weekly users: 0.07% showing signs of psychosis or mania (560,000), 0.15% showing heightened emotional attachment (1.2 million), 0.15% expressing suicidal ideation (1.2 million).
[3]John McCarthy et al., “A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence,” August 31, 1955.
[4]Harry Frankfurt, On Bullshit (Princeton: Princeton University Press, 2005). Originally published in Raritan Quarterly Review, 1986.
[5]Keith Sakata, psychiatry resident at UCSF, quoted in Bloomberg Businessweek, November 2025.