Deep sequential neural network models improve stratification of suicide attempt risk among US veterans

Abstract

Objective: To apply deep neural networks (DNNs) to longitudinal EHR data in order to predict suicide attempt risk among veterans. Local explainability techniques were used to provide explanations for each prediction with the goal of ultimately improving outreach and intervention efforts.

Materials and methods: The DNNs fused demographic information with diagnostic, prescription, and procedure codes. Models were trained and tested on EHR data of approximately 500 000 US veterans: all veterans with recorded suicide attempts from April 1, 2005, through January 1, 2016, each paired with 5 veterans of the same age who did not attempt suicide. Shapley Additive Explanation (SHAP) values were calculated to provide explanations of DNN predictions.

Results: The DNNs outperformed logistic and linear regression models in predicting suicide attempts. After adjusting for the sampling technique, the convolutional neural network (CNN) model achieved a positive predictive value (PPV) of 0.54 for suicide attempts within 12 months by veterans in the top 0.1% risk tier. Explainability methods identified meaningful subgroups of high-risk veterans as well as key determinants of suicide attempt risk at both the group and individual level.

Discussion and conclusion: The deep learning methods employed in the present study have the potential to significantly enhance existing suicide risk models for veterans. These methods can also provide important clues to explore the relative value of long-term and short-term intervention strategies. Furthermore, the explainability methods utilized here could also be used to communicate to clinicians the key features which increase specific veterans' risk for attempting suicide.

Keywords: Million Veteran Program; deep learning; electronic health records; suicide; veteran health.

Figure 1.

Study design: For each veteran who attempted suicide during the study period, we obtained 5 years of EHR data represented as 20 quarterly bins (top panel). In order to train the machine learning models to stratify veterans by the immediacy of their suicide attempt risk, we randomly selected 17 consecutive quarters of EHR data to include. Veterans whose suicide attempt occurred in the 3 months immediately following the last included quarter received a label of 1. Veterans whose suicide attempt occurred 9-12 months after the last included quarter received a label of 0.625, as shown in the middle panel. Omitted quarterly bins are colored white. Finally, we paired each veteran who attempted suicide with 5 veterans of similar age who did not attempt suicide during the study period (bottom panel); these veterans received a label of 0. Since fewer than 1% of veterans attempt suicide, we must undersample the negative examples (veterans without a suicide attempt) to train the models effectively, and the 5:1 ratio is more tractable than 99:1.

Figure 2.

DNN Model Architecture. The RNN model consists of two dense layers applied to each of the diagnosis, procedure, and prescription tables to create dense embeddings. Embeddings are concatenated with demographic information and passed through two sequential LSTM layers. The final output of the last LSTM layer is passed through two dense layers which produce a single final prediction. The CNN model consists of 1D-convolution layers that stride across the 17 time bins, each bin with fully concatenated data. The final convolutional output is flattened and passed through four dense layers to produce a single prediction.

Figure 3.

Risk score density: (A) Test set estimated score density as determined by the CNN model. An ideal result would score all positive cases (colored lines) with values near one and negative cases (black) with values near zero. (B) Both models struggle with the class imbalance, but the CNN model does a superior job of separating veterans at extreme risk (top 1.0% of scores). The CNN model separates high-risk individuals by time to event, as indicated by the score densities represented by colored lines.

Figure 4.

Individual veteran SHAP values: SHAP values for features with 10 highest positive contributions to veteran risk score as assigned by CNN model for 2 veterans who made a suicide attempt. (A) The CNN predicts a high-risk score for veteran A based on features associated with paraplegia, among others. (B) The CNN’s prediction for Veteran B relies heavily on mental health conditions and medications. (C) The CNN predicted that Veteran C was at high risk for a suicide attempt, but Veteran C did not attempt suicide. The features that contributed to the high-risk score are similar to those observed in veterans who did attempt suicide.

Figure 5.

SHAP and feature clustering: Clustering by SHAP values: (A) displays veteran subgroups identified the CNN model and produces higher quality clusters than those generated using raw patient data (feature values, B). Subgroups may demonstrate distinct trajectories that lead veterans towards suicide.