Development and temporal validation of a clinical prediction model of transition to psychosis in individuals at ultra-high risk in the UHR 1000+ cohort

Abstract

The concept of ultra-high risk for psychosis (UHR) has been at the forefront of psychiatric research for several decades, with the ultimate goal of preventing the onset of psychotic disorder in high-risk individuals. Orygen (Melbourne, Australia) has led a range of observational and intervention studies in this clinical population. These datasets have now been integrated into the UHR 1000+ cohort, consisting of a sample of 1,245 UHR individuals with a follow-up period ranging from 1 to 16.7 years. This paper describes the cohort, presents a clinical prediction model of transition to psychosis in this cohort, and examines how predictive performance is affected by changes in UHR samples over time. We analyzed transition to psychosis using a Cox proportional hazards model. Clinical predictors for transition to psychosis were investigated in the entire cohort using multiple imputation and Rubin's rule. To assess performance drift over time, data from 1995-2016 were used for initial model fitting, and models were subsequently validated on data from 2017-2020. Over the follow-up period, 220 cases (17.7%) developed a psychotic disorder. Pooled hazard ratio (HR) estimates showed that the Comprehensive Assessment of At-Risk Mental States (CAARMS) Disorganized Speech subscale severity score (HR=1.12, 95% CI: 1.02-1.24, p=0.024), the CAARMS Unusual Thought Content subscale severity score (HR=1.13, 95% CI: 1.03-1.24, p=0.009), the Scale for the Assessment of Negative Symptoms (SANS) total score (HR=1.02, 95% CI: 1.00-1.03, p=0.022), the Social and Occupational Functioning Assessment Scale (SOFAS) score (HR=0.98, 95% CI: 0.97-1.00, p=0.036), and time between onset of symptoms and entry to UHR service (log transformed) (HR=1.10, 95% CI: 1.02-1.19, p=0.013) were predictive of transition to psychosis. UHR individuals who met the brief limited intermittent psychotic symptoms (BLIPS) criteria had a higher probability of transitioning to psychosis than those who met the attenuated psychotic symptoms (APS) criteria (HR=0.48, 95% CI: 0.32-0.73, p=0.001) and those who met the Trait risk criteria (a first-degree relative with a psychotic disorder or a schizotypal personality disorder plus a significant decrease in functioning during the previous year) (HR=0.43, 95% CI: 0.22-0.83, p=0.013). Models based on data from 1995-2016 displayed good calibration at initial model fitting, but showed a drift of 20.2-35.4% in calibration when validated on data from 2017-2020. Large-scale longitudinal data such as those from the UHR 1000+ cohort are required to develop accurate psychosis prediction models. It is critical to assess existing and future risk calculators for temporal drift, that may reduce their utility in clinical practice over time.

Keywords: UHR 1000+ cohort; Ultra‐high risk for psychosis; brief limited intermittent psychotic symptoms; disorganized speech; negative symptoms; prediction model calibration; prediction of transition to psychosis; psychosocial functioning; temporal validation; unusual thought content.

Figure 1

Candidate predictors of transition to psychosis (hazard ratios with 95% CIs). CAARMS – Comprehensive Assessment of At‐Risk Mental States, BPRS – Brief Psychiatric Rating Scale, SANS – Scale for the Assessment of Negative Symptoms, SOFAS – Social and Occupational Functioning Assessment Scale, UHR – ultra‐high risk state, BLIPS – brief limited intermittent psychotic symptoms, APS – attenuated psychotic symptoms.

Figure 2

Change in performance over time for transition to psychosis prediction models developed in samples from 1995 to 2007 and from 1995 to 2016. A) Each plot displays the internal calibration curve using bootstrapping (N=1,000) and the temporal validation curve using individuals who presented at an UHR service between 2017 and 2020. The black diagonal line indicates an ideal calibration where the predicted probabilities match the observed probabilities. B) Each plot displays the decision curve analysis for the internal and temporal sample. Net benefit in the decision curve analysis is equivalent to true positive cases (i.e., a net benefit of 0.10 would be equivalent of identifying 10 individuals per 100, all of whom will transition to psychosis). A net benefit of zero (black horizontal line) is achieved when no individuals are treated.