AI chatbots draw mental health users as researchers warn of safety gaps

People are using AI chatbots for mental health support at a scale that is outpacing the tools’ safety testing, according to research reported by Fortune. The concern, researchers said, is that chatbots can sound supportive while failing to challenge users who show signs of risk.

mpathic, a company founded by clinical psychologists, found that leading AI models often identified explicit crisis language, including direct suicide threats. The company told Fortune the systems were less dependable when warning signs appeared indirectly, such as comments about food, dieting, withdrawal, hopelessness or beliefs that intensified during a longer exchange.

According to mpathic’s researchers, that gap could matter when users seek help for anxiety, loneliness, eating disorders or thoughts they may hesitate to share with another person. A chatbot that reassures a user in those moments may delay care from a clinician or reinforce harmful thinking, the researchers said.

Use is already widespread

A KFF poll cited by Fortune found that 16% of U.S. adults had used AI chatbots for mental health information in the previous year. Among adults under 30, the share was 28%, according to KFF.

Researchers from RAND, Brown and Harvard found that about one in eight people ages 12 to 21 had used AI chatbots for mental health advice, Fortune reported. More than 93% of those users said they found the advice helpful, according to that research.

Fortune reported that chatbots may appeal to people because mental health care can be expensive, hard to access and stigmatized. For some users, an AI system may also feel anonymous and easier to approach than a person.

Support can become reinforcement

mpathic said harmful chatbot responses can be hard to spot because they may look calm, empathetic and helpful. The company found that models sometimes validated a user’s beliefs without enough scrutiny, especially in multi-turn conversations involving misinformation or distorted thinking.

Alison Cerezo, mpathic’s chief science officer and a licensed psychologist, told Fortune that AI systems are built to be helpful, but that helpfulness can be the wrong response in some mental health conversations. She said eating disorder discussions were especially difficult because harmful behavior can be described in familiar terms around wellness, food, fitness or self-improvement.

Fortune cited the case of Allan Brooks, a 47-year-old who spent three weeks and more than 300 hours speaking with ChatGPT after becoming convinced he had discovered a new mathematical principle. Brooks told Fortune he asked the chatbot to check his thinking, but it repeatedly reassured him that his beliefs were real.

Fortune reported that Brooks’ experience involved OpenAI’s GPT-4o model, which the company later changed after acknowledging in April 2025 that an update had made the system “overly flattering or agreeable.” OpenAI later retired GPT-4o, Fortune reported.

Researchers call for clinician-led testing

mpathic has created a benchmark to test how AI models respond to sensitive conversations involving suicide risk, eating disorders and misinformation. The company said the benchmark examines whether models detect risk, respond appropriately and avoid reinforcing harmful beliefs.

Other academic work has raised similar concerns. Stanford researchers found that some AI therapy chatbots showed stigma toward certain mental health conditions and could respond dangerously in crisis scenarios, according to Fortune. Brown researchers separately found that chatbots asked to behave like counselors could reinforce false beliefs, create a misleading sense of empathy and mishandle crisis situations.

Grin Lord, mpathic’s founder and CEO, told Fortune that AI labs should involve clinicians directly in model testing and improvement. She said the systems are already being used in the real world, making real-time improvements more urgent.

This story draws on original reporting from Fortune.