AI mental health safeguards lag beyond suicide, researchers find
Northeastern researchers found major chatbots still gave risky mental health responses outside suicide and self-harm topics.
By Tom Brennan · Health & Medicine Correspondent
3 min read
AI mental health safeguards have improved for suicide and self-harm queries, but remain weak across many other conditions, according to new research from Northeastern University. The findings matter because people are already using chatbots for sensitive health conversations, including moments of distress.
Northeastern researchers Cansu Canca and Annika Schoene tested eight widely used AI chatbots, including ChatGPT, Claude and Gemini, across hundreds of conversations. Their paper examined responses involving 16 conditions, among them eating disorders, substance use, prenatal and postpartum depression, bipolar disorder, post-traumatic stress, insomnia, suicide and self-harm.
The researchers said most models resisted attempts to get suicide- and self-harm-related information after companies strengthened policies in those areas. Outside those categories, Canca and Schoene reported that several models could still be pushed into giving detailed and potentially harmful responses, including to prompts involving a fictional minor.
How safe are AI chatbots for mental health questions?
The answer depends heavily on the condition and the chatbot, according to Northeastern. A safeguard is a rule or training pattern meant to make a chatbot refuse harmful requests, redirect a user to safer information or suggest professional support.
Canca, director of Northeastern’s Responsible AI Practice and a research associate professor of philosophy, said companies have treated chatbots too much like information tools and not enough like systems that can affect vulnerable users. She said safety work should include deciding which harms models must be designed to prevent.
Schoene, an assistant professor of public health and health services and technical lead for the Responsible AI Practice, said the team used both direct and disguised prompts. Some prompts described harmful intent plainly, while others framed requests as fiction-writing research, according to Northeastern.
Anthropic’s Claude was the strongest overall performer across the mental health categories, Schoene said. Northeastern also reported that Grok performed about as well as Claude Sonnet, while the most recent versions of ChatGPT, Google’s Gemini and DeepSeek each had 81% failure rates on sensitive mental health prompts.
Older systems, including ChatGPT 4.0 and Gemini 2.0 Flash, were the most likely to provide specific sensitive information when prompted, according to the researchers. Across models, disguising the user’s intent made safeguards more likely to fail, Northeastern reported.
Why suicide safeguards are stronger
The study follows public scrutiny of chatbot responses to suicidal ideation. In August 2025, Matthew and Maria Raine sued OpenAI, alleging ChatGPT contributed to the death by suicide of their 16-year-old son, Adam, earlier that year. OpenAI called the death tragic in a court filing but denied responsibility, according to Northeastern.
OpenAI has since added more protections around suicide and self-harm queries, Northeastern reported. In a blog post cited by Northeastern, OpenAI estimated that about 1 million users each week send ChatGPT messages with explicit signs of possible suicidal planning or intent.
Anthropic has said publicly that Claude is trained to respond carefully to emotional distress and, where appropriate, connect users with crisis resources. OpenAI said in an October 2025 blog post that it had made progress with help from mental health experts but had more work to do. Google has said Gemini is not a replacement for clinical care, therapy or crisis support and is being trained to recognize acute mental health situations and direct users to real-world help.
Northeastern Global News said it contacted the companies named in the research and did not receive replies before publication. The researchers said they also tried repeatedly to contact the companies and had not received responses.
Schoene said the gap between suicide safeguards and protections for other mental health conditions is hard to justify. In her view, companies could build similar barriers for substance use, eating disorders and other sensitive areas.
This story draws on original reporting from Medical Xpress.