Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges

Abstract

Recent years have seen the rapid proliferation of clinical prediction models aiming to support risk stratification and individualized care within psychiatry. Despite growing interest, attempts to synthesize current evidence in the nascent field of precision psychiatry have remained scarce. This systematic review therefore sought to summarize progress towards clinical implementation of prediction modeling for psychiatric outcomes. We searched MEDLINE, PubMed, Embase, and PsychINFO databases from inception to September 30, 2020, for English-language articles that developed and/or validated multivariable models to predict (at an individual level) onset, course, or treatment response for non-organic psychiatric disorders (PROSPERO: CRD42020216530). Individual prediction models were evaluated based on three key criteria: (i) mitigation of bias and overfitting; (ii) generalizability, and (iii) clinical utility. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to formally appraise each study's risk of bias. 228 studies detailing 308 prediction models were ultimately eligible for inclusion. 94.5% of developed prediction models were deemed to be at high risk of bias, largely due to inadequate or inappropriate analytic decisions. Insufficient internal validation efforts (within the development sample) were also observed, while only one-fifth of models underwent external validation in an independent sample. Finally, our search identified just one published model whose potential utility in clinical practice was formally assessed. Our findings illustrated significant growth in precision psychiatry with promising progress towards real-world application. Nevertheless, these efforts have been inhibited by a preponderance of bias and overfitting, while the generalizability and clinical utility of many published models has yet to be formally established. Through improved methodological rigor during initial development, robust evaluations of reproducibility via independent validation, and evidence-based implementation frameworks, future research has the potential to generate risk prediction tools capable of enhancing clinical decision-making in psychiatric care.

Fig. 1

PRISMA flow diagram of study selection.

Fig. 2. Geographic coverage of reviewed study samples.

A total of 21 of the 228 studies (9.2%) incorporated data from several countries (range: 2–24) via multi-national samples or research consortia and thus are counted multiple times, yielding a total count of 369. World map coordinates retrieved from maps and ggplot2 packages in R.

Fig. 3. Psychiatric outcomes among prediction models.

A Distribution of psychiatric diagnoses across individual prediction models (n = 308). B Distribution of psychiatric diagnoses across deciles of mean sample age, where data were available (n = 246). ADHD attention-deficit/hyperactivity disorder, ASD autism spectrum disorder, BPD borderline personality disorder, OCD obsessive-compulsive disorder, ODD oppositional defiant disorder, PTSD post-traumatic stress disorder, SUDs substance use disorders. ‘Mixed affective’ models tested some combination of anxiety, depressive, and/or manic symptoms in a single outcome, while ‘transdiagnostic’ models consolidated several externalizing and/or internalizing diagnoses. Due to rounding, percentages may not add up to 100%.

Fig. 4. Risk of bias among prediction models.

Domain-level summary of risk of bias for all developed prediction models within the reviewed literature (n = 308). For individual PROBAST domain ratings across all development and external validation analyses, see Supplementary Table S3.

Fig. 5. Internal and external validation efforts and performance among prediction models.

A Summary of internal validation methods among all prediction models (n = 308). B Overlap between internal and external validation efforts. C Distribution of c-indices/AUCs in the development sample (with or without internal validation) and external validation sample for models where both were reported (n = 49). Blue lines denote superior performance in the development sample, while red lines indicate superior external performance. Where a model has been validated in more than one external sample, an average c-index/AUC across these samples has been derived.