Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Jan 31:10:100109.
doi: 10.1016/j.xjmad.2025.100109. eCollection 2025 Jun.

Applying large language models to stratify suicide risk using narrative clinical notes

Thomas H McCoy  1   2 , Roy H Perlis  1   2
Affiliations

Applying large language models to stratify suicide risk using narrative clinical notes

Thomas H McCoy et al. J Mood Anxiety Disord. .

Abstract

We investigated whether large language models can stratify risk for suicide following hospital discharge. We drew on a very large cohort of 458,053 adults discharged from two academic medical centers between January 4, 2005 and January 2, 2014, linked to administrative vital status data. From this sample, each of the 1995 individuals who died by suicide or accident was matched with 5 control individuals on the basis of age, sex, race and ethnicity, admitting hospital, insurance, comorbidity index, and discharge year. We applied a HIPAA-compliant large language model (gpt-4-1106-preview) to estimate risk for suicide based on narrative discharge summaries. In the resulting cohort (n = 11,970), median age was 57 (IQR 44 -76); 4536 (38 %) were women; 348 (3 %) had a primary psychiatric admission diagnosis. For the model-predicted risk, time to 90 % survival was 1588 days (IQR 1374-1905) in the lowest-risk quartile, 1432 (IQR 1157-1651) in the 2nd quartile, 661 (IQR 538-820) in the 3rd quartile, and 302 (IQR 260-362) in the top quartile (p < .001). In Fine and Gray competing risk regression, predicted hazard was significantly associated with observed risk (unadjusted HR 7.66 [95 % CI 6.40-9.27]; adjusted for sociodemographic features and utilization, HR 8.86 (7.00-11.2)). Estimated risks were significantly greater scores among individuals who were Black or Hispanic (p < .005 for each, versus white individuals). Overall, a large language model (LLM) was able to stratify risk for suicide and accidental death among individuals discharged from academic medical centers beyond that afforded by simple sociodemographic and clinical features medical centers.

Keywords: Accident; Artificial Intelligence; Machine learning; Mortality; Self-harm; Suicide.

PubMed Disclaimer

Conflict of interest statement

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests Roy Perlis reports financial support was provided by National Institutes of Health. Roy Perlis reports a relationship with Genomind Inc that includes: consulting or advisory. Roy Perlis reports a relationship with Circular Genomics that includes: consulting or advisory and equity or stocks. Roy Perlis reports a relationship with Alkermes Inc that includes: consulting or advisory. Roy Perlis reports a relationship with Psy Therapeutics that includes: consulting or advisory and equity or stocks. Dr. Perlis is Editor in Chief of JAMA + AI and an associate editor for JAMA Network Open but was not involved in the editorial review or decision process for this manuscript. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Figures

Fig. 1
Fig. 1
Kaplan-Meier survival curves of death by suicide or accident, by quartile of model-estimated risk.
Fig. 2
Fig. 2
A. Calibration of regression model of risk for death by suicide or accident, incorporating sociodemographic features, utilization, and model-estimated risk. Note: Only individuals (n = 11, 296) for whom adequate follow-up was available were incorporated to examine 3-year outcomes. Fig. 2B. Re-calibration of regression model of risk for death by suicide or accident, incorporating sociodemographic features, utilization, and model-estimated risk.
See this image and copyright information in PMC

References

    1. Perlis R.H., Fihn S.D. Hard truths about suicide prevention. JAMA Netw Open. 2020;3(10) - PubMed
    1. Belsher B.E., Smolenski D.J., Pruitt L.D., Bush N.E., Beech E.H., Workman D.E., et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76(6):642–651. - PubMed
    1. Simon G.E., Johnson E., Lawrence J.M., Rossom R.C., Ahmedani B., Lynch F.L., et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951–960. doi: 10.1176/appi.ajp.2018.17101167. - DOI - PMC - PubMed
    1. Kessler R.C., Warner C.H., Ivany C., Petukhova M.V., Rose S., Bromet E.J., et al. Predicting suicides after psychiatric hospitalization in US army soldiers: the army study to assess risk and resilience in servicemembers (Army STARRS) JAMA Psychiatry. 2015;72(1):49–57. - PMC - PubMed
    1. McCoy T.H., Castro V.M., Roberson A.M., Snapper L.A., Perlis R.H. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry. 2016;73(10):1064. doi: 10.1001/jamapsychiatry.2016.2172. - DOI - PMC - PubMed

LinkOut - more resources