Prognostic predictions in psychosis: exploring the complementary role of machine learning models

Abstract

Background: Predicting outcomes in schizophrenia spectrum disorders is challenging due to the variability of individual trajectories. While machine learning (ML) shows promise in outcome prediction, it has not yet been integrated into clinical practice. Understanding how ML models (MLMs) can complement psychiatrists' predictions and bridge the gap between MLM capabilities and practical use is key.

Objective: This vignette study aims to compare the performance of psychiatrists and MLMs in predicting short-term symptomatic and functional remission in patients with first-episode psychosis and explore whether MLMs can improve psychiatrists' prognostic accuracy.

Method: 24 psychiatrists predicted symptomatic and functional remission probabilities at 10 weeks based on written baseline information from 66 patients in the OPtimization of Treatment and Management of Schizophrenia in Europe (OPTiMiSE) trial. ML-generated predictions based on these vignettes were then shared with psychiatrists, allowing them to adjust their estimates.

Findings: The predictive accuracy of the MLM was low but comparable to that of psychiatrists for symptomatic remission (MLM: 0.50, psychiatrists: 0.52) and comparable to that of psychiatrists for functional remission (MLM: 0.72, psychiatrists: 0.79). Inter-rater agreement was low but comparable for psychiatrists and the MLM. Although the MLM did not improve overall predictive accuracy, it showed potential in aiding psychiatrists with difficult-to-predict cases. However, psychiatrists struggled to recognise when to rely on the model's output, and we were unable to determine a clear pattern in these cases based on their characteristics.

Conclusions: MLMs may have the potential to support psychiatric decision-making, particularly in difficult-to-predict cases, but at present, their effectiveness remains limited due to constraints in predictive accuracy and the ability to identify when to rely on the model's output. Addressing these issues is crucial to improve the utility of MLMs and foster their integration into clinical practice.

Clinical implications: MLMs are best suited as supplementary tools, providing a second opinion while psychiatrists retain decision-making autonomy. Integrating predictions from both sources may help reduce individual biases and improve accuracy. This approach leverages the strengths of MLMs without compromising clinical responsibility.

Keywords: Machine Learning; Schizophrenia & psychotic disorders.

Figure 1. The 66 cases are grouped based on the number of psychiatrists who correctly predicted them. The blue bars represent the number of cases in each group (‘occurrence’), while the blue line indicates psychiatrist accuracy (number of correct predictions divided by 8), and the orange line shows MLM accuracy. For both symptomatic and functional remission, MLM accuracy is higher than psychiatrist accuracy in the ‘hard cases’ (≤3 correct predictions by psychiatrists). The difference in accuracy was (just) not statistically significant, with a p value of 0.06 for both symptomatic and functional remission. MLM, machine learning model; PSY, psychiatrists.