Dynamic prediction of psychological treatment outcomes: development and validation of a prediction model using routinely collected symptom data

Abstract

Background: Common mental disorders can be effectively treated with psychotherapy, but some patients do not respond well and require timely identification to prevent treatment failure. We aimed to develop and validate a dynamic model to predict psychological treatment outcomes, and to compare the model with currently used methods, including expected treatment response models and machine learning models.

Methods: In this prediction model development and validation study, we obtained data from two UK studies including patients who had accessed therapy via Improving Access to Psychological Therapies (IAPT) services managed by ten UK National Health Service (NHS) Trusts between March, 2012, and June, 2018, to predict treatment outcomes. In study 1, we used data on patient-reported depression (Patient Health Questionnaire 9 [PHQ-9]) and anxiety (Generalised Anxiety Disorder 7 [GAD-7]) symptom measures obtained on a session-by-session basis (Leeds Community Healthcare NHS Trust dataset; n=2317) to train the Oracle dynamic prediction model using iterative logistic regression analysis. The outcome of interest was reliable and clinically significant improvement in depression (PHQ-9) and anxiety (GAD-7) symptoms. The predictive accuracy of the model was assessed in an external test sample (Cumbria Northumberland Tyne and Wear NHS Foundation Trust dataset; n=2036) using the area under the curve (AUC), positive predictive values (PPVs), and negative predictive values (NPVs). In study 2, we retrained the Oracle algorithm using a multiservice sample (South West Yorkshire Partnership NHS Foundation Trust, North East London NHS Foundation Trust, Cheshire and Wirral Partnership NHS Foundation Trust, and Cambridgeshire and Peterborough NHS Foundation Trust; n=42 992) and compared its performance with an expected treatment response model and five machine learning models (Bayesian updating algorithm, elastic net regularisation, extreme gradient boosting, support vector machine, and neural networks based on a multilayer perceptron algorithm) in an external test sample (Whittington Health NHS Trust; Barnet Enfield and Haringey Mental Health Trust; Pennine Care NHS Foundation Trust; and Humber NHS Foundation Trust; n=30 026).

Findings: The Oracle algorithm trained using iterative logistic regressions generalised well to external test samples, explaining up to 47·3% of variability in treatment outcomes. Prediction accuracy was modest at session one (AUC 0·59 [95% CI 0·55-0·62], PPV 0·63, NPV 0·61), but improved over time, reaching high prediction accuracy (AUC 0·81 [0·77-0·86], PPV 0·79, NPV 0·69) as early as session seven. The performance of the Oracle model was similar to complex (eg, including patient profiling variables) and computationally intensive machine learning models (eg, neural networks based on a multilayer perceptron algorithm, extreme gradient boosting). Furthermore, the predictive accuracy of a more simple dynamic algorithm including only baseline and index-session scores was comparable to more complex algorithms that included additional predictors modelling sample-level and individual-level variability. Overall, the Oracle algorithm significantly outperformed the expected treatment response model (mean AUC 0·80 vs 0·70, p<0·0001]).

Interpretation: Dynamic prediction models using sparse and readily available symptom measures are capable of predicting psychotherapy outcomes with high accuracy.

Funding: University of Sheffield.