Generalizability of clinical prediction models in mental health
- PMID: 40108256
- PMCID: PMC12240828
- DOI: 10.1038/s41380-025-02950-0
Generalizability of clinical prediction models in mental health
Abstract
Concerns about the generalizability of machine learning models in mental health arise, partly due to sampling effects and data disparities between research cohorts and real-world populations. We aimed to investigate whether a machine learning model trained solely on easily accessible and low-cost clinical data can predict depressive symptom severity in unseen, independent datasets from various research and real-world clinical contexts. This observational multi-cohort study included 3021 participants (62.03% females, MAge = 36.27 years, range 15-81) from ten European research and clinical settings, all diagnosed with an affective disorder. We firstly compared research and real-world inpatients from the same treatment center using 76 clinical and sociodemographic variables. An elastic net algorithm with ten-fold cross-validation was then applied to develop a sparse machine learning model for predicting depression severity based on the top five features (global functioning, extraversion, neuroticism, emotional abuse in childhood, and somatization). Model generalizability was tested across nine external samples. The model reliably predicted depression severity across all samples (r = 0.60, SD = 0.089, p < 0.0001) and in each individual external sample, ranging in performance from r = 0.48 in a real-world general population sample to r = 0.73 in real-world inpatients. These results suggest that machine learning models trained on sparse clinical data have the potential to predict illness severity across diverse settings, offering insights that could inform the development of more generalizable tools for use in routine psychiatric data analysis.
© 2025. The Author(s).
Conflict of interest statement
Competing interests: FP is a member of the European Scientific Advisory Board of Brainsway Inc., Jerusalem, Israel, and the International Scientific Advisory Board of Sooma, Helsinki, Finland. He has received speaker’s honoraria from Mag&More GmbH and the neuroCare Group. His lab has received support with equipment from neuroConn GmbH, Ilmenau, Germany, and Mag&More GmbH and Brainsway Inc., Jerusalem, Israel. MAR has received financial research support from the EU (H2020 No. 754740) and served as PI in clinical trials from Abide Therapeutics, Böhringer-Ingelheim, Emalex Biosciences, Lundbeck GmbH, Nuvelution TS Pharma Inc., Oryzon, Otsuka Pharmaceuticals and Therapix Biosciences. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Consent to participate: Written informed consent was obtained from all participants of this study. Study approval: The research in all samples was conducted ethically in accordance with the World Medical Association Declaration of Helsinki and approved by all samples’ local institutional review boards and ethics committees named hereafter: • Ethikkommission beider Basel (EKBB), • Klinisches Ethikkomitee der Universitären Psychiatrischen Kliniken Basel (KLINEK UPK), • Ethics Commission of the Westphalia-Lippe Medical Association and the University of Münster, • Ethics Committee of the Faculty of Psychology and Sports Science at the University of Münster, • Ethics Commission of the Faculty of Medicine at the University of Münster, • Ethics Commission of the Faculty of Medicine at the Philipps-University Marburg, • Ethics Commission of the Faculty of Medicine of Friedrich-Schiller-University Jena, • Ethics Commission of the Faculty of Medicine at Ludwig-Maximilians-University Munich, • Ethics Commission of the Faculty of Medicine at Martin-Luther-University Halle-Wittenberg, • Kommission für ethische Fragen der Wissenschaft der Martin-Luther-Universität Halle-Wittenberg (KeFW), • Ethics Commission of the Faculty of Medicine of Cologne University, • Ethics Committee of the Medical Faculty at Heinrich Heine University of Düsseldorf, • Ethics Commission of the University of Turku, • Independent Ethical Committee of the University of Bari Aldo Moro (Il Comitato Etico), • Departmental Commission for the Experimentation and Protection of the Person of the Department of Medicine of the University of Udine (IRB-DMED), • Ethics Committee of the University of Milan, • Ethics Committee of the University of Birmingham.
Figures
