Evaluating the clinical utility of an easily applicable prediction model of suicide attempts, newly developed and validated with a general community sample of adults

Abstract

Background: A suicide attempt (SA) is a clinically serious action. Researchers have argued that reducing long-term SA risk may be possible, provided that at-risk individuals are identified and receive adequate treatment. Algorithms may accurately identify at-risk individuals. However, the clinical utility of algorithmically estimated long-term SA risk has never been the predominant focus of any study.

Methods: The data of this report stem from CoLaus|PsyCoLaus, a prospective longitudinal study of general community adults from Lausanne, Switzerland. Participants (N = 4,097; M_age = 54 years, range: 36-86; 54% female) were assessed up to four times, starting in 2003, approximately every 4-5 years. Long-term individual SA risk was prospectively predicted, using logistic regression. This algorithm's clinical utility was assessed by net benefit (NB). Clinical utility expresses a tool's benefit after having taken this tool's potential harm into account. Net benefit is obtained, first, by weighing the false positives, e.g., 400 individuals, at the risk threshold, e.g., 1%, using its odds (odds of 1% yields 1/(100-1) = 1/99), then by subtracting the result (400*1/99 = 4.04) from the true positives, e.g., 5 individuals (5-4.04), and by dividing the result (0.96) by the sample size, e.g., 800 (0.96/800). All results are based on 100 internal cross-validations. The predictors used in this study were: lifetime SA, any lifetime mental disorder, sex, and age.

Results: SA at any of the three follow-up study assessments was reported by 1.2%. For a range of seven a priori selected threshold probabilities, ranging between 0.5% and 2%, logistic regression showed highest overall NB in 97.4% of all 700 internal cross-validations (100 for each selected threshold probability).

Conclusion: Despite the strong class imbalance of the outcome (98.8% no, 1.2% yes) and only four predictors, clinical utility was observed. That is, using the logistic regression model for clinical decision making provided the most true positives, without an increase of false positives, compared to all competing decision strategies. Clinical utility is one among several important prerequisites of implementing an algorithm in routine practice, and may possibly guide a clinicians' treatment decision making to reduce long-term individual SA risk. The novel metric NB may become a standard performance measure, because the a priori invested clinical considerations enable clinicians to interpret the results directly.

Keywords: Adult; Algorithm; Clinical utility; Decision support; Net benefit; Suicide attempt.

Fig. 1

Median net benefit among 100 test subsamples, based on the resampling procedure, which was the same for each of the seven selected threshold probabilities. The solid line shows the logistic regression model, and the dot-dashed line shows the classification and regression tree (CART) model. The harm-to-benefit ratios are rounded to one decimal place (Inf = infinity, expressing an infinite harm to missing any true positive subject in the entire population)