A generalizable deep learning regression model for automated glaucoma screening from fundus images

Abstract

A plethora of classification models for the detection of glaucoma from fundus images have been proposed in recent years. Often trained with data from a single glaucoma clinic, they report impressive performance on internal test sets, but tend to struggle in generalizing to external sets. This performance drop can be attributed to data shifts in glaucoma prevalence, fundus camera, and the definition of glaucoma ground truth. In this study, we confirm that a previously described regression network for glaucoma referral (G-RISK) obtains excellent results in a variety of challenging settings. Thirteen different data sources of labeled fundus images were utilized. The data sources include two large population cohorts (Australian Blue Mountains Eye Study, BMES and German Gutenberg Health Study, GHS) and 11 publicly available datasets (AIROGS, ORIGA, REFUGE1, LAG, ODIR, REFUGE2, GAMMA, RIM-ONEr3, RIM-ONE DL, ACRIMA, PAPILA). To minimize data shifts in input data, a standardized image processing strategy was developed to obtain 30° disc-centered images from the original data. A total of 149,455 images were included for model testing. Area under the receiver operating characteristic curve (AUC) for BMES and GHS population cohorts were at 0.976 [95% CI: 0.967-0.986] and 0.984 [95% CI: 0.980-0.991] on participant level, respectively. At a fixed specificity of 95%, sensitivities were at 87.3% and 90.3%, respectively, surpassing the minimum criteria of 85% sensitivity recommended by Prevent Blindness America. AUC values on the eleven publicly available data sets ranged from 0.854 to 0.988. These results confirm the excellent generalizability of a glaucoma risk regression model trained with homogeneous data from a single tertiary referral center. Further validation using prospective cohort studies is warranted.

Fig. 1. Overview of the G-RISK regression approach versus conventional glaucoma detection CNNs that are trained with binary labels.

Both models were described in our previous work on explainable AI for glaucoma detection. The mismatch between the prevalence in a tertiary referral center (used for model development) and sparse real world data (external testing) leads to overprediction in the latter. The prediction histogram illustrates this phenomenon in the binary classification approach (a), with significantly more cases referred to as being glaucomatous than with G-RISK (b). Also note the spike in cases with prediction close to 1, versus a consistent decrease in cases as the prediction value increases for G-RISK. TV refers to the optimal threshold value. TV is typically fixed at 0.5 in binary classification models due to a sharp sigmoid/softmax activation function. In a regression approach with linear activation, TV can be set at a different value, depending on the costs associated with FP and FN. c Examples of fundus images with an increasing G-RISK score.

Fig. 2. Examples of the training set and thirteen data sets used for external testing of G-RISK for generalizable glaucoma detection.

Each pair displays a randomly selected original unprocessed image that features glaucoma-induced damage (left) and the corresponding 30° disc-centered result after image manipulation (right), prepared for G-RISK input.

Fig. 3. Combined ROC curve, calibration curve, and prediction histogram plots per data set that featured an available glaucoma ground truth.

The top plot area features (1) the ROC curve (light green) with false positive rate and true positive rate on the x and y axis, (2) as well as the calibration curve (dark green) with mean predicted value and the fraction of positives on the x and y axis. A diagonal dotted black line between (0,0) and (1,1) indicates the ROC curve of random prediction and optimal calibration. The vertically flipped histogram of G-RISK predictions is aligned with the calibration curve in the bottom plot, with prediction value on the x axis, and prediction count on the y axis. Best viewed in color.

Fig. 4. Overview of top three most extreme false-positive cases (three first images from the left per row) and false-negative cases (three rightmost images per row) per evaluated data set (name printed in the left corner of the first image per row).

GHS data was left out as there exists no ground truth on image level. The predicted risk score is at the bottom right for each image. Best viewed in color and high resolution for optimal review by the reader. See Supplementary Fig. 3 for a view with overlaid saliency map.