Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Oct 1;7(10):e2439269.
doi: 10.1001/jamanetworkopen.2024.39269.

Performance of Machine Learning Suicide Risk Models in an American Indian Population

Emily E Haroz  1   2 , Paul Rebman  2 , Novalene Goklish  1 , Mitchell Garcia  1 , Rose Suttle  1 , Dominick Maggio  3 , Eben Clattenburg  3 , Joe Mega  3 , Roy Adams  4
Affiliations

Performance of Machine Learning Suicide Risk Models in an American Indian Population

Emily E Haroz et al. JAMA Netw Open. .

Abstract

Importance: Few suicide risk identification tools have been developed specifically for American Indian and Alaska Native populations, even though these populations face the starkest suicide-related inequities.

Objective: To examine the accuracy of existing machine learning models in a majority American Indian population.

Design, setting, and participants: This prognostic study used secondary data analysis of electronic health record data collected from January 1, 2017, to December 31, 2021. Existing models from the Mental Health Research Network (MHRN) and Vanderbilt University (VU) were fitted. Models were compared with an augmented screening indicator that included any previous attempt, recent suicidal ideation, or a recent positive suicide risk screen result. The comparison was based on the area under the receiver operating characteristic curve (AUROC). The study was performed in partnership with a tribe and local Indian Health Service (IHS) in the Southwest. All patients were 18 years or older with at least 1 encounter with the IHS unit during the study period. Data were analyzed between October 6, 2022, and July 29, 2024.

Exposures: Suicide attempts or deaths within 90 days.

Main outcomes and measures: Model performance was compared based on the ability to distinguish between those with a suicide attempt or death within 90 days of their last IHS visit with those without this outcome.

Results: Of 16 835 patients (mean [SD] age, 40.0 [17.5] years; 8660 [51.4%] female; 14 251 [84.7%] American Indian), 324 patients (1.9%) had at least 1 suicide attempt, and 37 patients (0.2%) died by suicide. The MHRN model had an AUROC value of 0.81 (95% CI, 0.77-0.85) for 90-day suicide attempts, whereas the VU model had an AUROC value of 0.68 (95% CI, 0.64-0.72), and the augmented screening indicator had an AUROC value of 0.66 (95% CI, 0.63-0.70). Calibration was poor for both models but improved after recalibration.

Conclusion and relevance: This prognostic study found that existing risk identification models for suicide prevention held promise when applied to new contexts and performed better than relying on a combined indictor of a positive suicide risk screen result, history of attempt, and recent suicidal ideation.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: None reported.

Figures

Figure.
Figure.. Receiver Operating Characteristic (ROC) Curves for Mental Health Research Network (MHRN) Primary Care, Vanderbilt University (VU), and Existing Screen for 90-Day Suicide Attempts for 331 558 Patient Visits
PPV indicates positive predictive value.
See this image and copyright information in PMC

References

    1. Barlow A, Haroz EE, O’Keefe VM, et al. . New collaborative research on suicide prevention, practice, and policy with American Indian and Alaska Native communities holds promise for all peoples. Health Promot Pract. 2023;24(5):841-851. doi:10.1177/15248399221116630 - DOI - PMC - PubMed
    1. Karaye IM. Differential trends in US suicide rates, 1999-2020: emerging racial and ethnic disparities. Prev Med. 2022;159:107064. doi:10.1016/j.ypmed.2022.107064 - DOI - PubMed
    1. Stone DM, Mack KA, Qualters J. Notes from the field: recent changes in suicide rates, by race and ethnicity and age group—United States, 2021. MMWR Morb Mortal Wkly Rep. 2023;72(6):160-162. doi:10.15585/mmwr.mm7206a4 - DOI - PMC - PubMed
    1. Gone JP, Trimble JE. American Indian and Alaska Native mental health: diverse perspectives on enduring disparities. Annu Rev Clin Psychol. 2012;8:131-160. doi:10.1146/annurev-clinpsy-032511-143127 - DOI - PubMed
    1. Haroz EE, Walsh CG, Goklish N, Cwik MF, O’Keefe V, Barlow A. Reaching those at highest risk for suicide: development of a model using machine learning methods for use with Native American communities. Suicide Life Threat Behav. 2020;50(2):422-436. doi:10.1111/sltb.12598 - DOI - PMC - PubMed

Publication types