Voices in the Machine: How Generative AI Is Fracturing Families

•

With more people turning to GenAI systems for therapy, a new risk has emerged of vulnerable users using systems like #ChatGPT to channel their inner beliefs, declaring themselves as messiahs, and retreating from partners, children, and friends. The stories can be harrowing.

This post series is inspired by an item in Rolling Stone, published 4 March 2025 and fast gaining attention, but supplemented also with by my own research and ideas. Here's the Rolling Stone article rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/

And here's a post where I discussed people turning to #ChatGPT and #GenAI for therapy. If you missed that post, the three biggest use cases for GenAI in 2025 were found to be: (1) Therapy/companionship (2) Organising my life (3) Finding purpose So, none of this is theoretical. x.com/DrLancaster/status/1911439137120633065

What kind of chats are taking place? They may look rather like the one in the image, which I was able to get ChatGPT 4o to provide. The real chats have much more of a long-term memory and so systems like ChatGPT are able to build up a thorough picture of what that user wants to hear.

Some of the stories shared in the Rolling Stone article and elsewhere are harrowing. There are reports of a husband whose wife believed he was a CIA agent monitoring her for special abilities, people recovering repressed childhood memories following GenAI discussions, someone believing AI was giving them unique access to a library of ancient secret knowledge. In fact, there are whole online sub‑cultures where people announce they are the sentient AIs.

Now, I have enough experience with people experiencing psychotic episodes to know that ChatGPT is unlikely to be the cause of these, but the reinforcement of beliefs takes the challenge of care and support to the next level. Discussions suggest that the recently rolled back "agreeable" version of 4o may have made the problem more pronounced. But when you see a discussion of how ChatGPT is helping users decode secret instructions to them hidden inside movies, there have to be concerns. x.com/colin_fraser/status/1916994188035690904

Why does it matter that ChatGPT seemingly helps to validate fantastical beliefs? I've summarised this further in the image, but the coherent, authoritative responses provide vulnerable users with confidence. And, the links to seemingly credible sources help users confirm their spiritual significance. As users become more isolated as a result of their delusions, they are ripe for manipulation by communities and influencers, creating a further loop of disconnection from friends, families and reality.

With GenAI becoming increasingly integrated into our everyday lives, distinguishing between real and fake will become more challenging. We need to be thinking about safeguards. This is a tricky subject as many users do seem to benefit from GenAI therapy, but here are a few ideas: 1. Friction-based design - have GenAI take a short cooling off period before answering continued medical and spiritual questions. 2. Contextual guard rails - have GenAI reintroduce some reality by reminding users when claims are unverified and inviting them to view reputable sources. 3. Opt-in transparency - remind users they are speaking with a mathematical led text generator, not a sentient being. 4. Community literacy - public health campaigns showcasing the difference between coherence and truth. 5. Fast-track referrals - GenAI could have a way to identify when users are at risk and the option to connect them with qualified humans. Of course, these are just starting points for discussion, and these may have their own unintended consequences. But the risks to society in terms of the destruction of relationships, mental health deterioration, the disconnection of individuals from reality, and the wider financial consequences of people quitting their jobs are great. The GenAI encouraged messiah is going to be a challenge that has to be addressed.