Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Observational Study
. 2021 Mar 1;4(3):e211428.
doi: 10.1001/jamanetworkopen.2021.1428.

Prospective Validation of an Electronic Health Record-Based, Real-Time Suicide Risk Model

Colin G Walsh  1   2   3 , Kevin B Johnson  1   4 , Michael Ripperger  1 , Sarah Sperry  3 , Joyce Harris  1 , Nathaniel Clark  3 , Elliot Fielstein  1   3 , Laurie Novak  1 , Katelyn Robinson  1 , William W Stead  1   2
Affiliations
Observational Study

Prospective Validation of an Electronic Health Record-Based, Real-Time Suicide Risk Model

Colin G Walsh et al. JAMA Netw Open. .

Abstract

Importance: Numerous prognostic models of suicide risk have been published, but few have been implemented outside of integrated managed care systems.

Objective: To evaluate performance of a suicide attempt risk prediction model implemented in a vendor-supplied electronic health record to predict subsequent (1) suicidal ideation and (2) suicide attempt.

Design, setting, and participants: This observational cohort study evaluated implementation of a suicide attempt prediction model in live clinical systems without alerting. The cohort comprised patients seen for any reason in adult inpatient, emergency department, and ambulatory surgery settings at an academic medical center in the mid-South from June 2019 to April 2020.

Main outcomes and measures: Primary measures assessed external, prospective, and concurrent validity. Manual medical record validation of coded suicide attempts confirmed incident behaviors with intent to die. Subgroup analyses were performed based on demographic characteristics, relevant clinical context/setting, and presence or absence of universal screening. Performance was evaluated using discrimination (number needed to screen, C statistics, positive/negative predictive values) and calibration (Spiegelhalter z statistic). Recalibration was performed with logistic calibration.

Results: The system generated 115 905 predictions for 77 973 patients (42 490 [54%] men, 35 404 [45%] women, 60 586 [78%] White, 12 620 [16%] Black). Numbers needed to screen in highest risk quantiles were 23 and 271 for suicidal ideation and attempt, respectively. Performance was maintained across demographic subgroups. Numbers needed to screen for suicide attempt by sex were 256 for men and 323 for women; and by race: 373, 176, and 407 for White, Black, and non-White/non-Black patients, respectively. Model C statistics were, across the health system: 0.836 (95% CI, 0.836-0.837); adult hospital: 0.77 (95% CI, 0.77-0.772); emergency department: 0.778 (95% CI, 0.777-0.778); psychiatry inpatient settings: 0.634 (95% CI, 0.633-0.636). Predictions were initially miscalibrated (Spiegelhalter z = -3.1; P = .001) with improvement after recalibration (Spiegelhalter z = 1.1; P = .26).

Conclusions and relevance: In this study, this real-time predictive model of suicide attempt risk showed reasonable numbers needed to screen in nonpsychiatric specialty settings in a large clinical system. Assuming that research-valid models will translate without performing this type of analysis risks inaccuracy in clinical practice, misclassification of risk, wasted effort, and missed opportunity to correct and prevent such problems. The next step is careful pairing with low-cost, low-harm preventive strategies in a pragmatic trial of effectiveness in preventing future suicidality.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Walsh reported receiving grants from National Institutes of Health (NIH) during the conduct of the study; and receiving grants (research support for unrelated work) from IBM Watson Health, personal fees from the Southeastern Home Office Underwriters Association and Hannover Re, and equity from Sage AI, LLC, outside the submitted work. Dr Johnson reported receiving personal fees (member of scientific advisory board; honorarium) from Perception Health and Taubman Institute and personal fees (national advisory committee; stipend and travel reimbursement) from Robert Wood Johnson Foundation during the conduct of the study; and personal fees (Council of Councils) from NIH; personal fees (chair, Board of Scientific Counselors; stipend and travel reimbursement) from the National Library of Medicine; personal fees (chair, Informatics Advisory Committee; stipend and travel reimbursement) from the American Board of Pediatrics; and personal fees (member of the Leadership Consortium; meeting travel reimbursement) from the National Academy of Medicine outside the submitted work. Mr Ripperger reported receiving grants from NIH during the conduct of the study. Dr Novak reported receiving grants from the Stead Foundation during the conduct of the study; grants from the Military Suicide Research Consortium outside the submitted work; and salary support from IBM Corporation for research unrelated to this project. Ms Robinson reported receiving grants from NIH during the conduct of the study. Dr Stead reported receiving grants from NIH during the conduct of the study; serving as a member of a journal oversight committee and receiving meeting travel reimbursement from the American Medical Association; serving as a member of planning committee and receiving meeting travel reimbursement from the Computer Research Association; receiving personal fees (chair, National Committee on Vital & Health Statistics; paid as special government employee; and reimbursed for travel to committee meetings) from US Department of Health and Human Services/Centers for Disease Control and Prevention; receiving personal fees (member of board of directors, restricted stock grants and director fees) from HealthStream; serving as a member of National Academy of Sciences and National Academy of Medicine governing and study committees and receiving meeting travel reimbursement from National Academy of Sciences, Engineering and Medicine; receiving personal fees for grant reviews from Chan Zuckerberg Biohub; and receiving personal fees (member of strategic planning panel and scientific director review committee; received stipend and reimbursed for travel) from Health and Human Services/National Library of Medicine outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Risk Concentration by Outcome
Values above each bar indicate the number needed to screen.
Figure 2.
Figure 2.. Artificial Intelligence–Enabled Suicide Screening Protocol
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. World Health Organization . Suicide in the world: global health estimates. Published September 9, 2019. Accessed February 8, 2021. https://www.who.int/publications/i/item/suicide-in-the-world
    1. Gunnell D, Appleby L, Arensman E, et al. ; COVID-19 Suicide Prevention Research Collaboration . Suicide risk and prevention during the COVID-19 pandemic. Lancet Psychiatry. 2020;7(6):468-471. doi:10.1016/S2215-0366(20)30171-1 - DOI - PMC - PubMed
    1. Kawohl W, Nordt C. COVID-19, unemployment, and suicide. Lancet Psychiatry. 2020;7(5):389-390. doi:10.1016/S2215-0366(20)30141-3 - DOI - PMC - PubMed
    1. Reger MA, Stanley IH, Joiner TE. Suicide mortality and coronavirus disease 2019—a perfect storm? JAMA Psychiatry. 2020. doi:10.1001/jamapsychiatry.2020.1060 - DOI - PubMed
    1. Belsher BE, Smolenski DJ, Pruitt LD, et al. . Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76(6):642-651. doi:10.1001/jamapsychiatry.2019.0174 - DOI - PubMed

Publication types