AI for Suicide Prevention: Examining the potential and risks of AI in suicidal crisis support
Large language models are increasingly used by the public for mental health-related questions, including moments of acute suicidal crisis. Yet general-purpose AI systems are not specifically developed, clinically validated, or systematically evaluated for this highly sensitive context. This creates both a promising opportunity and a major responsibility: AI may offer immediate, scalable support in moments of crisis, but inappropriate responses may also carry substantial risks.AI for Suicide Prevention examines how artificial intelligence can be responsibly developed and evaluated for patient-facing suicidal crisis support. The project starts by evaluating how currently available general-purpose large language models respond to suicidal crisis situations, with a particular focus on clinical appropriateness, safety, and potentially harmful response patterns.Building on these findings, the project will develop and optimize an AI-supported crisis-support system specifically designed for acute suicidal crisis support and responsible use in mental health contexts. AI-generated crisis-support responses will first be assessed for clinical appropriateness and safety using standardized suicidal crisis scenarios evaluated by clinicians and individuals with lived experience. In a subsequent experimental study, the optimized crisis-support system will be compared with general-purpose large language models in simulated crisis conversations with LLM-based patient agents. In a later phase, the system will be examined in a pilot feasibility study with patients directly within psychiatric care. The project is conducted in cooperation with the TUM Klinikum, Department of Psychiatry and Psychotherapy, and the Chair of Health Informatics. The project aims to inform the responsible development and clinical evaluation of patient-facing AI tools for acute suicidal crisis support