Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative

Abstract

Purpose: Advances in artificial intelligence have enabled the development of predictive models for glaucoma. However, most work is single-center and uncertainty exists regarding the generalizability of such models. The purpose of this study was to build and evaluate machine learning (ML) approaches to predict glaucoma progression requiring surgery using data from a large multicenter consortium of electronic health records (EHR).

Design: Cohort study.

Participants: Thirty-six thousand five hundred forty-eight patients with glaucoma, as identified by International Classification of Diseases (ICD) codes from 6 academic eye centers participating in the Sight OUtcomes Research Collaborative (SOURCE).

Methods: We developed ML models to predict whether patients with glaucoma would progress to glaucoma surgery in the coming year (identified by Current Procedural Terminology codes) using the following modeling approaches: (1) penalized logistic regression (lasso, ridge, and elastic net); (2) tree-based models (random forest, gradient boosted machines, and XGBoost), and (3) deep learning models. Model input features included demographics, diagnosis codes, medications, and clinical information (intraocular pressure, visual acuity, refractive status, and central corneal thickness) available from structured EHR data. One site was reserved as an "external site" test set (N = 1550); of the patients from the remaining sites, 10% each were randomly selected to be in development and test sets, with the remaining 27 999 reserved for model training.

Main outcome measures: Evaluation metrics included area under the receiver operating characteristic curve (AUROC) on the test set and the external site.

Results: Six thousand nineteen (16.5%) of 36 548 patients underwent glaucoma surgery. Overall, the AUROC ranged from 0.735 to 0.771 on the random test set and from 0.706 to 0.754 on the external test site, with the XGBoost and random forest model performing best, respectively. There was greatest performance decrease from the random test set to the external test site for the penalized regression models.

Conclusions: Machine learning models developed using structured EHR data can reasonably predict whether glaucoma patients will need surgery, with reasonable generalizability to an external site. Additional research is needed to investigate the impact of protected class characteristics such as race or gender on model performance and fairness.

Financial disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Keywords: Deep learning; Glaucoma; Machine learning; Multicenter study.

Figure 3

Deep learning model architecture. Depiction of the architecture of deep learning model predicting which patients would progress to surgery within 12 months. This model passes the medication and International Classification of Diseases (ICD) code features through an embedding layer.

Figure 4

Receiver operating characteristic curves and precision-recall curves for machine learning and deep learning prediction models. This figure depicts receiver operating characteristic curves and precision-recall curves for models predicting glaucoma progression to surgery. A, Curves for machine learning models. B, Curves for deep learning models. All models were evaluated on the test set (comprising individuals from from the same sites as the training set) and an external test site (comprising individuals from a site that was not included in the training set). LASSO = least absolute shrinkage and selection operator.

Figure 5

Model explainability with Shapley feature importance. The figure depicts the Shapley value for the top most important features for predicting whether a patient would progress to the point of requiring surgery within the next year, using the XGBoost model and calculated across the test set. Points represent individual observations (patients) in the test. The feature value color of each point indicates whether the value of that feature was high or low for that individual observation. A positive Shapley value for a feature for an individual point indicates influence toward a model prediction of surgery, whereas a negative Shapley value indicates influence toward a model prediction of no surgery. ICD = International Classification of Diseases; IOP = intraocular pressure; logMAR = logarithm of the minimum angle of resolution; OD = right eye; OS = left eye; SHAP = SHapley Additive exPlanations.