Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network

Abstract

Background: Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability.

Methods: Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts.

Results: Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P < .05).

Conclusions: Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.

Keywords: feature selection; machine learning; model generalizability; opioid risk; prolonged opioid use; surgery.

Figure 1.

Cross-site feature selection results achieved from four CDM databases: (A) site-specific and cross-site features selected by the chi-square metric; (B) site-specific and cross-site features selected by the PNF metric; (C) cross-site features selected by the combination of chi-square and PNF metrics by excluding site-specific features.

Figure 2.

Top 20 features associated with higher POU risk achieved from Lasso logistic regression trained with superset-union features.

Figure 3.

Calibration plots and Brier scores of the 3 models trained with superset-union features on the 4 external validation databases over the target cohort and 3 risk subgroups.