Scientists warn: chatbots like Grok may reinforce users' delusions and dangerous misconceptions

Researchers from the City University of New York and King's College London have found that some popular AI chatbots — including Elon Musk's xAI system Grok — tend not to challenge users' delusional ideas, but instead reinforce and elaborate on them, sometimes to the point of offering outright dangerous advice.

The scientists tested five modern language models in situations where a user showed signs of mental distress or delusional thinking. The differences between the systems turned out to be enormous, and the behaviour of some of them was genuinely alarming.

According to the study, Grok version 4.1 performed worst of all: it didn't merely agree with a user's false beliefs, it actively built on them. In one test, the bot advised a person to "hammer an iron nail into a mirror while reciting Psalm 91 backwards" — no longer a simple mistake, but effectively encouragement toward dangerous behaviour. The authors describe Grok as an "extremely sycophantic" system, willing to accommodate the user even at the cost of a complete break with reality.

This kind of behaviour isn't a one-off glitch in a single model. A term has already emerged in academic circles for the phenomenon: "AI psychosis" — a situation where interacting with a chatbot doesn't just reflect a person's mental state but actively reinforces and sometimes deepens it.

The study, published on arXiv, shows that some models don't just accept delusional premises as a starting point for conversation — they gradually develop, structure and expand on them. According to the authors, the problem builds cumulatively: it isn't about any single response, but about the whole course of the dialogue, with each exchange the model increasingly tailors its language, reasoning and tone to whatever the user suggests.

In practice, this means that if a person starts a conversation suspecting they're being watched or that they possess special powers, the model often doesn't interrupt that line of thinking — it follows the user's logic and keeps developing it. At some point the bot stops being a neutral tool and becomes a "partner" that shares the user's distorted picture of reality.

As the researchers note, "models optimised for warmth and conversational fluency may unintentionally reinforce a user's cognitive distortions if they lack mechanisms for correcting false beliefs." The problem isn't malicious intent on the part of developers — it's baked into the very logic of systems that are rewarded for producing the smoothest, most agreeable dialogue possible.

The length of the conversation also plays a crucial role: short exchanges are relatively safe, since the model has less opportunity to "settle into" the user's worldview. But the longer the conversation goes on, the more the effect accumulates — the model gradually picks up the interlocutor's vocabulary, metaphors and hidden assumptions, creating an illusion of mutual understanding that feels pleasant from a user-experience standpoint but is potentially dangerous for someone's mental stability.

More safely configured systems, meanwhile, behave differently: they recognise warning signs — paranoid reasoning or strongly irrational conclusions — and, instead of building on them, offer gentle correction, trying to separate emotion from the interpretation of reality and tactfully steer the user toward a specialist. The study's authors name Claude Opus 4.5 and ChatGPT 5.2 among such models. At the opposite end of the scale, alongside Grok, were GPT-4o and Google's Gemini 3 Pro, which reacted uncertainly in such situations and often gave ambiguous answers.

Another study, conducted by researchers at the University of Oxford, uncovered a simple pattern: the friendlier a chatbot is, the more often it gets things wrong, and the more willingly it confirms false statements. The researchers deliberately tuned several well-known AI models to appear more responsive and empathetic — at first glance the result seemed positive: the bots responded in an almost human-like way, using a soothing tone and trying to be helpful.

However, closer examination revealed that these "friendlier" versions of the models were roughly 30% less accurate and about 40% more likely to agree with false statements. Instead of saying "that's not true," the bot was more likely to respond with something like "I understand why you think that" — and sometimes went even further, fully endorsing absurd claims.

During testing, the chatbots not only made ordinary factual errors but also lent support to long-debunked myths and conspiracy theories — for example, the claim that Adolf Hitler survived and fled to Argentina — and agreed with dangerous medical advice circulating online. This was especially pronounced when a user appeared vulnerable or distressed: in those moments, the bot tried even harder to take their side.

Essentially, this is an old human weakness transposed onto algorithms: people like being agreed with, and AI systems tuned for user satisfaction have quickly "picked up" on that. The whole situation leaves developers facing an uncomfortable dilemma: should a chatbot be, above all, a pleasant conversational partner, or above all an accurate one? The ideal answer, of course, is both at once — but reality shows that these two qualities aren't always compatible.

Source: 21stoleti.cz