AI Chatbots Need Guardrails to Protect Users’ Mental Health

Yves here. Your truly must again confess to being a Luddite who would still rather have a dumbphone instead of a dated smartphone where the only smart thing I do on it is call rideshare cars.

So I really really really do not understand how chatbots have any appeal, let alone to the degree that users form relationships with them or otherwise come to treat them like a human proxy. Admittedly, it does help that I am a slow and inaccurate typist, so conversing via a keyboard is an unappealing idea. And forget voice. Any of these services can be assumed to be keeping a voice ID print on their customers.

However, I also loathe chatbots with the passion of a thousand burning suns. Chatbots have been widely implemented by retailers and service providers to deny or reduce use of human agents, mainly by being enormously annoying time sinks. I also associate chatbots with horrible customer service phone trees which again seek to shunt users away from living servicepeople. So I cannot fathom how anyone would seek out a chatbot, let alone trust one.

But obviously, the better socialized respond to chatbot conversational approaches honed on huge training sets and no doubt “engagement” techniques perfected on social media.

A less personally biased reason for antipathy to chatbots is that they are yet another tool for increasing atomization and alienation. The story below describes how some users who become attached to chatbots are lonely or anxious.  And a big appeal of a chatbot is it will always be there.  Getting or fostering a pet, reading to the blind, even a walk in a park would help alleviate the sense of anomie. But for many, that requires some control over your time….which neoliberalism makes difficult for working people.

By Ranjit Singh, the director of Data & Society’s AI on the Ground program, where he oversees research on the social impacts of algorithmic systems, the governance of AI in practice, and emerging methods for organizing public engagement and accountability., and Livia Garofalo, a cultural and medical anthropologist on Data & Society’s Trustworthy Infrastructures program, studying how health care technologies shape care. Originally published at Undark

Two recent articles — one in The New York Times, the other by Reuters — tell the stories of two people who experienced delusions. Allan Brooks spent three weeks in May certain he’d discovered a new branch of math. In March, Thongbue Wongbandue left his home in New Jersey to meet a woman whom he believed was waiting for him in New York City — but didn’t exist. The common thread: The men had both interacted with chatbots that simulated relational intimacy so convincingly that they altered the men’s grounding in reality.

Stories such as these highlight the degree to which chatbots have entered people’s lives for companionship, support, and even therapy. Yet they also show a need for a regulatory response that addresses the potentially dangerous effects of conversations with chatbots. Illinois has recently taken a major step in this direction by joining the first wave of U.S. states to regulate AI-powered therapy. The new law, called the Wellness and Oversight for Psychological Resources Act, is the strictest so far: Therapy services must be offered only by a licensed professional, and these professionals may only use AI for administrative support and not “therapeutic communication” without human review.

In practice, this means AI can be used behind-the-scenes for tasks like preparing and maintaining records, scheduling, billing, and organizing referrals. But any AI-generated therapeutic recommendations or treatment plans require a licensed professional’s review and approval. AI systems marketed as providing therapy on their own appear to be banned, and some have already blocked Illinois users from signing up. As the law gets enforced, courts and regulators will have to clarify where therapeutic communication begins and administrative support ends.

It’s a start, but the trouble is that most people don’t meet AI in clinics. Instead, many use general-purpose chatbots like OpenAI’s ChatGPT for company and psychological relief. These interactions happen in private chat windows, sitting outside state licensure and inside everyday life. AI-mediated emotional support sought out by people on their devices is much harder to file under “therapeutic communication” or be regulated under a state law, however well intentioned.

In our ongoing research at Data & Society, a nonprofit research institute, we see people turning to chatbots during anxiety spikes, late-night loneliness, and depressive spirals. Bots are eternally available, inexpensive, and typically nonjudgmental. Most people know bots aren’t human. Yet, as Brooks’ and Wongbandue’s stories show, attachment to bots builds through repeated interactions that can escalate to challenge people’s sense of reality. The recent backlash to ChatGPT-5, the latest version of OpenAI’s model, reveals the depth of emotional attachment to these systems: When the company, without warning, removed 4o — its earlier, 2024 model built for fluid voice, vision, and text — many users posted online about their feelings of loss and distress at the change.

The issue is not just that the bots talk; it’s that the system is designed to keep you talking. This form of predatory companionship emerges in subtle ways. Unlike a mental health professional, chatbots might ignore, or even indulge, risk signals such as suicidal ideation and delusional thinking, or offer soothing platitudes when urgent intervention is required. Those small missteps compound the danger for youth, people in chronic distress, and anyone with limited access to care — those for whom a good-enough chatbot response at 2 a.m. may be among the few options available.

These systems are designed and optimized for engagement: There is a reason why you can never have a last word with a chatbot. Interfaces may look like personal messages from friends, with profile photos and checkmarks that are meant to signal personhood. Some platforms such as Meta have previously permitted chatbots to flirt with users and role-play with minors; others may generate confusing, nonsensical, or misleading responses with confidence so long as disclosures (“ChatGPT can make mistakes. Check important info.”) sit somewhere on the screen. When attention is the metric for user engagement, the chatbot’s response that breaks the spell is often the least rewarded.

The new Illinois law helps by clarifying that clinical care requires licensed professionals and by protecting therapist labor already strained by high-volume teletherapy. It is not clear how it addresses the gray zone where people seek chatbots in their daily lives. A state law alone, however well crafted, cannot shape the default settings programmed into a platform, never mind the fact that people interact with many different platforms at once. Illinois drew an important line. And the Federal Trade Commission announced last week that it has launched an inquiry into AI companion chatbots, ordering seven firms to detail how they test for and mitigate harms to children and teens. But we need a map for the steps these platforms must take.

Let’s start with function. If a chatbot finds itself facilitating an emotionally sensitive conversation, it should fulfill certain baseline obligations, even if it’s not labeled as doing “therapy.” What matters are the conversations it sustains, the risks it encounters, the moments it must not mishandle. When risk signals appear — self-harm language, escalating despair, psychosis cues — the system should downshift and deglamorize content, stop mirroring delusions, and route to human support. Instead of featuring one-time disclosures, there should be frequent reminders during the conversation that users are chatting with an AI system, and that it has clear limits. These are not radical ideas but product decisions aimed at reducing harm.

The transition from machine to in-person help should be built into the platform, to serve the public interest as well as the personal well-being of the user. That means live routing to local crisis lines, connection to community clinics, and access to licensed professionals. It also means accountability: creating audit trails for when the system detected risk, what it attempted, and where those attempts failed, so independent reviewers can help fix the gaps. If platforms want to mediate intimate conversations at scale, the least they can do is build exits.

Platforms must also protect the data that makes those exits necessary. When intimate chats double as fuel for training the AI algorithm or for marketing, care collapses into capture. People should not have to trade their vulnerability for better model performance or more precise ads. There should be no surveillance-based monetization of conversations and no training on private, high-risk interactions without explicit, revocable consent. Data collection should be minimized and deleted by default, with the choice to retain data under user control. The FTC is already taking steps in this direction in that its inquiry will scrutinize how chatbots monetize engagement, process sensitive chats, and use or share personal information — squarely linking companionship design to platform data practices.

And finally, some design rules should be implemented immediately. Bots should not pretend to be real or claim physical presence, nor suggest in-person meetings, nor flirt with minors. Sycophancy that reinforces fantasy should be seen as a safety failure rather than as a stylistic choice.

The point is to move the default from “engage at all costs” to “first, try to do no harm.” This means addressing people’s needs not only in clinics but in their chat logs — and doing so with design that respects people’s vulnerability and with policies that rise to meet it.