Evaluating machine learning classifiers for glaucoma referral decision support in primary care settings

Abstract

Several artificial intelligence algorithms have been proposed to help diagnose glaucoma by analyzing the functional and/or structural changes in the eye. These algorithms require carefully curated datasets with access to ocular images. In the current study, we have modeled and evaluated classifiers to predict self-reported glaucoma using a single, easily obtained ocular feature (intraocular pressure (IOP)) and non-ocular features (age, gender, race, body mass index, systolic and diastolic blood pressure, and comorbidities). The classifiers were trained on publicly available data of 3015 subjects without a glaucoma diagnosis at the time of enrollment. 337 subjects subsequently self-reported a glaucoma diagnosis in a span of 1-12 years after enrollment. The classifiers were evaluated on the ability to identify these subjects by only using their features recorded at the time of enrollment. Support vector machine, logistic regression, and adaptive boosting performed similarly on the dataset with F1 scores of 0.31, 0.30, and 0.28, respectively. Logistic regression had the highest sensitivity at 60% with a specificity of 69%. Predictive classifiers using primarily non-ocular features have the potential to be used for identifying suspected glaucoma in non-eye care settings, including primary care. Further research into finding additional features that improve the performance of predictive classifiers is warranted.

Figure 1

The average Precision-Recall curves for all classifiers with respect to a dummy classifier. The area under the curve (AUC) reported as mean (standard deviation): Adaptive boosting (AdaBoost) – 0.30 (0.07), support vector machine – 0.29 (0.05), and logistic regression – 0.28 (0.05).

Figure 2

Permutation feature importance applied to each classifier: (a) Logistic regression, (b) Support vector machine, and (c) Adaptive boosting (AdaBoost). Mean decrease in F1 score is shown for each feature: age, systolic and diastolic blood pressure, gender (male), body mass index (BMI), intraocular pressure (IOP) in the right eye (RE) and left eye (LE), age-related macular degeneration (AMD) category, race (black, Hispanic, Asian, and other), and presence of diabetes and arthritis.