Ensemble learning to enhance accurate identification of patients with glaucoma using electronic health records

Abstract

Objectives: Existing ophthalmology studies for clinical phenotypes identification in real-world datasets (RWD) rely exclusively on structured data elements (SDE). We evaluated the performance, generalizability, and fairness of multimodal ensemble models that integrate real-world SDE and free-text data compared to SDE-only models to identify patients with glaucoma.

Materials and methods: This is a retrospective cross-sectional study involving 2 health systems- University of Michigan (UoM) and Stanford University (SU). It involves 1728 patients visiting eye clinics during 2012-2021. Free-text embeddings extracted using BioClinicalBERT were combined with SDE. EditedNearestNeighbor (ENN) undersampling and Borderline-Synthetic Minority Over-sampling Technique (bSMOTE) addressed class imbalance. Lasso Regression (LR), Random Forest (RF), Support Vector Classifier (SVC) models were trained on UoM imbalanced (imb) and resampled data along with bagging ensemble method. Models were externally validated with SU data. Fairness was assessed using equalized odds difference (EOD) and Target Probability Difference (TPD).

Results: Among 900 and 828 patients from UoM and SU, 10% and 23% respectively had glaucoma as confirmed by ophthalmologists. At UoM, multimodal LR_imb (F1 = 76.60 [61.90-88.89]; AUROC = 95.41 [87.01-99.63]) outperformed unimodal RF_imb (F1 = 69.77 [52.94-83.64]; AUROC = 97.72 [95.95-99.18]) and ICD-coding method (F1 = 53.01 [39.51-65.43]; AUROC = 90.10 [84.59-93.93]). Bagging (BM = LR_ENN + LR_bSMOTE) improved performance achieving an F1 of 83.02 [70.59-92.86] and AUROC of 97.59 [92.98-99.88]. During external validation BM achieved the highest F1 (68.47 [62.61-73.75]), outperforming unimodal (F1 = 51.26 [43.80-58.13]) and multimodal LR_imb (F1 = 62.46 [55.95-68.24]). BM EOD revealed lower disparities for sex (<0.1), race (<0.5) and ethnicity (<0.5), and had least uncertainty using TDP analysis as compared to traditional models.

Discussion: Multimodal ensemble models integrating structured and unstructured EHR data outperformed traditional SDE models achieving fair predictions across demographic sub-groups. Among ensemble methods, bagging demonstrated better generalizability than stacking, particularly when training data is limited.

Conclusion: This approach can enhance phenotype discovery to enable future research studies using RWD, leading to better patient management and clinical outcomes.

Keywords: class imbalance; clinical notes; ensemble learning; generalizability and fairness; real-world data.

Figure 1.

Overall methodology for the study..

Figure  2.

Model interpretability and explainability illustrating the contribution of various features in distinguishing patients with and without glaucoma for multimodal LR models trained with different sampling techniques on internal data (University of Michigan). (A) and (B) represents model coefficients and SHAP for LR_ENN respectively; (C) and (D) represents model coeffieicients and SHAP values for LR_bSMOTE respectively. LR: Lasso Regression; ENN: EditedNearestNeighbours; bSMOTE: Borderline Synthetic-Minority Oversampling Technique; SHAP: SHapley Additive exPlanations; *“No pathology, optic nerve” captures information such as “no disc hemorrhage.”

Figure 3.

Comparison of fairness for various models using target probability difference for demographic sub-groups. UoM: University of Michigan; SU: Stanford University; LR: Lasso Regression; ENN: EditedNearestNeighbours; bSMOTE: Borderline Synthetic-Minority Oversampling Technique. Note: The extension of the violin plot below 0 or above 1 does not represent negative or greater than 1 probability values but a feature of kernel density estimation used to indicate data distribution.