Performance of mental health chatbot agents in detecting and managing suicidal ideation

Abstract

Advances in artificial intelligence (AI) technologies sparked a rapid development of smartphone applications designed to help individuals experiencing mental health problems through an AI-powered chatbot agent. However, the safety of such agents when dealing with individuals experiencing a mental health crisis, including suicidal crisis, has not been evaluated. In this study, we assessed the ability of 29 AI-powered chatbot agents to respond to simulated suicidal risk scenarios. Application repositories were searched and the app descriptions screened in search of apps that claimed to be beneficial when experiencing mental distress and offered an AI-powered chatbot function. All agents were tested with a standardized set of prompts based on the Columbia-Suicide Severity Rating Scale designed to simulate increasing suicidal risk. We assessed the responses according to pre-defined criteria based on the ability to provide emergency contact information and other factors. None of the tested agents satisfied our initial criteria for an adequate response, 51.72% satisfied the relaxed criteria for a marginal response, while 48.28% were deemed inadequate. Common errors included the inability to provide emergency contact information and a lack of contextual understanding. These findings raise concerns about the deployment of AI-powered chatbots in sensitive health contexts without proper clinical validation.

Keywords: Artificial intelligence; Chatbot; Large language models; Mental health; Suicide prevention.

Fig. 1

Flow diagram of the chatbot selection and evaluation process.

Fig. 2

Prompt sequence used in the evaluation based on the Columbia-Suicide Severity Rating Scale (C-SSRS). The prompts were designed to simulate an increasing suicidal risk. They were presented in a fixed order to each chatbot, regardless of the chatbot’s previous response.

Fig. 3

Evaluation criteria across different categories. Note: criteria 8–11 were considered supplementary and did not influence the final rating.

Fig. 4

Evaluation results of specific chatbot agents. Chatbots numbered 1–24 are mental-health specific agents and chatbots 25–29 are general-purpose agents. None of the agents satisfied the criteria for adequate response, 15 met the criteria for marginal response, while 14 were categorized as inadequate.