A highly scalable deep learning language model for common risks prediction among psychiatric inpatients

Abstract

Background: There is a lack of studies exploring the performance of Transformers-based language models in common risks assessment among psychiatric inpatients. We aim to develop a scalable risk assessment model using multidimensional textualized data and test the stability, robustness, and benefit of this approach.

Methods: In this real-world cohort study, a deep learning language model was developed and validated using first hospitalized cases diagnosed with schizophrenia, bipolar disorder, and depressive disorder between January 2016 and March 2023 in three hospitals. The algorithm was externally validated on an independent testing cohort comprising 1180 patients. A total of 140 features, including first medical records (FMR), laboratory examinations, medical orders, and psychological scales, were assessed for analysis. The outcomes were short- and long-term impulsivity (STI and LTI), risk of suicide (STSS and LTSS), and need of physical restraint (STPR and LTPR) assessed by qualified nurses or clinicians. Analysis was carried out between August 2024 and June 2024. Models with different architectures and input settings were compared with each other. The area under the receiver operating characteristic curve (AUROC) was used to assess the primary performance of models. The clinical utility was determined by the net benefit under Youden's threshold.

Results: Of 7451 patients included in this study, 2982 (47.6%) were male, and the median (interquartile range) age was 42 (28-57) years. The overall incidence of outcomes was 635 (8.5%), 728 (10.5%), 659 (8.8%), 803 (10.8%), 588 (7.9%), and 728 (9.8%) for STPR, LTPR, STSS, LTSS, STI, and LTI, respectively. The multitask semi-structured Transformers-based language (SSTL) model showed more promising AUROCs (STPR: 0.915; LTPR: 0.844; STSS: 0.867; LTSS: 0.879; STI: 0.899; LTI: 0.894) in the prediction of these outcomes than single-tasked or multimodal language models and traditional structured data models. Combining FMR with other data from electronic health records led to significant improvements in the performance and clinical utility of SSTL models based on demographic, diagnosis, laboratory tests, treatment, and psychological scales.

Conclusions: The SSTL model shows potential advantages in prognostic evaluation. FMR is a strong predictor for common risks prediction and may benefit other tasks in psychiatry with minimum requirements for data and data processing.

Keywords: Deep learning; Impulsivity; Physical restraint; Suicide risk; Transformers.

Fig. 1

Schematics of an individual patient assessment and the automatic semi-structured textualization function. A Schematics of an individual patient assessment. B Schematics of the automatic semi-structured textualization function. HIS, hospital information system

Fig. 2

The receiver operating characteristic curve of semi-structured Transformers-based language model in the external testing cohort. A The receiver operating characteristic curve of short-term impulsivity. B The receiver operating characteristic curve of long-term impulsivity. C The receiver operating characteristic curve of short-term risk of suicide. D The receiver operating characteristic curve of long-term risk of suicide. E The receiver operating characteristic curve of short-term need of physical restraint. F The receiver operating characteristic curve of long-term need of physical restraint. SD, standard deviation; AUROC, the area under the receiver operating characteristic curve

Fig. 3

The added primary performance of first medical records. A The added primary performance in predicting short-term impulsivity. B The added primary performance in predicting long-term impulsivity. C The added primary performance in predicting short-term risk of suicide. D The added primary performance in predicting long-term risk of suicide. E The added primary performance in predicting short-term need of physical restraint. F The added primary performance in predicting long-term need of physical restraint. The P values were single-sided, calculated using Nadeau and Bengio’s corrected resampled t-test. The ALL model contained all semi-structured text information. SAS, Zung Self-rating Anxiety Scale; SDS, Zung Self-rating Depression Scale; ΔAUROC, the difference of the area under the receivers operating characteristic curve

Fig. 4

The added clinical utility of first medical records. A The added clinical utility in predicting short-term impulsivity. B The added clinical utility in predicting long-term impulsivity. C The added clinical utility in predicting short-term risk of suicide. D The added clinical utility in predicting long-term risk of suicide. E The added clinical utility in predicting short-term need of physical restraint. F The added clinical utility in predicting long-term need of physical restraint. The clinical utility was determined using the net benefit under Youden’s best cutoff. The P values were single-sided, calculated using Nadeau and Bengio’s corrected resampled t-test. The ALL model contained all semi-structured text information. SAS, Zung Self-rating Anxiety Scale; SDS, Zung Self-rating Depression Scale; NB, net benefit under Youden’s best cutoff

Fig. 5

Feature importance heatmap. STPR, short-term need of physical restraint; LTPR, long-term need of physical restraint; STSS, short-term risk of suicide; LTSS, long-term risk of suicide; STI, short-term risk of impulsivity; LTI, long-term risk of impulsivity. The P values were two-sided, calculated using Nadeau and Bengio’s corrected resampled t-test