A deep learning system for predicting time to progression of diabetic retinopathy

Abstract

Diabetic retinopathy (DR) is the leading cause of preventable blindness worldwide. The risk of DR progression is highly variable among different individuals, making it difficult to predict risk and personalize screening intervals. We developed and validated a deep learning system (DeepDR Plus) to predict time to DR progression within 5 years solely from fundus images. First, we used 717,308 fundus images from 179,327 participants with diabetes to pretrain the system. Subsequently, we trained and validated the system with a multiethnic dataset comprising 118,868 images from 29,868 participants with diabetes. For predicting time to DR progression, the system achieved concordance indexes of 0.754-0.846 and integrated Brier scores of 0.153-0.241 for all times up to 5 years. Furthermore, we validated the system in real-world cohorts of participants with diabetes. The integration with clinical workflow could potentially extend the mean screening interval from 12 months to 31.97 months, and the percentage of participants recommended to be screened at 1-5 years was 30.62%, 20.00%, 19.63%, 11.85% and 17.89%, respectively, while delayed detection of progression to vision-threatening DR was 0.18%. Altogether, the DeepDR Plus system could predict individualized risk and time to DR progression over 5 years, potentially allowing personalized screening intervals.

Fig. 1. Design of the DeepDR Plus system.

a, Schematic overview of the DeepDR Plus system. DeepDR Plus can predict the time to DR progression and perform risk stratification using retinal images of individuals with diabetes. b, Evaluation and application of the AI system. We trained our AI system on a developmental dataset and tested the generalizability in eight longitudinal independent cohorts. c, Schematic overview of the real-world study. d, Visual diagram of the DeepDR Plus system. DM, diabetes mellitus; DR, diabetic retinopathy; NPDR, non-proliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy; DME, diabetic macular edema; DRPS, Diabetic Retinopathy Progression Study; ECHM, Eastern China Health Management; WTHM, Wuhan Tongji Health Management; NDSP, Nicheng Diabetes Screening Project; CUHK-STDR, Chinese University of Hong Kong-Sight-Threatening Diabetic Retinopathy; PUDM, Peking Union Diabetes Management; SEED, Singapore Epidemiology of Eye Diseases; SiDRP, Singapore National Diabetic Retinopathy Screening Program; BJHC, Beijing Healthcare Cohort Study; IM, integrated management; DL, deep learning. Source data

Fig. 2. Internal and external validation of the fundus model in the prediction of the progression of DR.

a, IBS (left) showing overall fit (lower is better) and C-index (right) measuring model risk discrimination (higher is better) for various time points. Data of external validation include retinal fundus images from individuals in the ECHM, WTHM, CUHK-STDR and PUDM cohorts. b, Kaplan–Meier plots for the prediction of DR progression. The x axis indicates the time in years. The y axis is the survival probability, measuring the probability of no DR progressing in 5 years. One-sided log-rank test was used for the comparison between the low-risk and high-risk groups. The P values for the internal test set and the external validation datasets 1, 2, 4 and 5 are 1.554 × 10⁻⁴¹, 3.258 × 10⁻⁴⁶, 4.867 × 10⁻¹⁷, 2.946 × 10⁻¹⁹ and 1.888 × 10⁻⁴, respectively. c, Prediction of DR progression using time-dependent ROC curves. The asterisk indicates that there is only one case of the progression from non-DR to DR in the first year. The shaded areas in a and b denote 95% CIs. Areas under the ROC curves are presented as mean values (lower bound of 95% CI, upper bound of 95% CI). Source data

Fig. 3. Explainability analysis of DeepDR Plus in predicting DR progression.

a, Comparisons of color fundus photographs at baseline and follow-up using attention maps. b, Mean attention maps and corresponding stack fundus images for any DR progression and subgroups 1–3. c, Bar plot (left) of fundus score and clinical features and their contribution to the prediction model of DR progression. Features are in descending order by contribution (also known as importance) in the model. Details of associations are shown in a beeswarm plot (right) in which each point represents a participant. Color indicates the value of the feature, with red denoting higher and blue denoting lower. A negative SHAP value indicates negative feature attribution for the prediction of DR progression; a positive SHAP value indicates positive feature attribution for the prediction of DR progression. Source data

Extended Data Fig. 1. Datasets flowchart.

DRPS, Diabetic Retinopathy Progression Study; ECHM, The Eastern China Health Management; WTHM, Wuhan Tongji Health Management; NDSP, Nicheng Diabetes Screening Project; CUHK-STDR, The Chinese University of Hong Kong-Sight-Threatening Diabetic Retinopathy; PUDM: Peking Union Diabetes Management; SEED, the Singapore Epidemiology of Eye Diseases study; SiDRP, the Singapore National Diabetic Retinopathy Screening Program; BJHC, Beijing Healthcare Cohort Study.

Extended Data Fig. 2. Model performance in predicting time to progression of eyes with DR progression in the internal test set and external validation dataset–1,2,4, and 5.

a, Bland–Altman plots for the agreement between the predicted and actual time to DR progression. The x axis represents the mean of predicted and actual time to DR progression (average DR progression time), and the y axis represents the difference between the two measurements. b, Box plots show the distribution of samples for the absolute error for three models (the fundus model, the metadata model and the combined model) (n = 859). The horizontal line indicates the median and the whiskers indicate the lowest and highest points within the interquartile ranges of the lower or upper quartile, respectively. Mann-Whitney U test was used for the comparison among the models. R², coefficient of determination; MAE, mean absolute error. Source data

Extended Data Fig. 3. Internal and external validation of the DeepDR Plus system in the prediction of the progression from non-DR to DR.

a, Integrated Brier score (left) showing overall fit—lower is better and C-index (right) measuring model risk discrimination—higher is better—for various time points. b, Kaplan–Meier plots for the prediction of the progression from non-DR to DR. One-sided log-rank test was used for the comparison between the low- and high-risk groups. The P values on internal test set and external validation dataset–1,2,4, and 5 are 6.638×10⁻³⁸, 2.181×10⁻⁴⁶, 5.453×10⁻¹⁶, 1.167×10⁻¹² and 2.508×10⁻³, respectively. c, Prediction of the progression from non-DR to DR using time-dependent ROC curves. *The 1-year ROC of External-5 where only one case of the progression from non-DR to DR occurred that year. Shaded areas in a and b are 95% CIs. Areas under ROC curves are presented as mean values (lower bound of 95% CI, upper bound of 95% CI). Source data

Extended Data Fig. 4. Internal and external validation of the DeepDR Plus system in the prediction of the progression from non-referable DR to referable DR.

a, Integrated Brier score (left) showing overall fit—lower is better and C-index (right) measuring model risk discrimination—higher is better—for various time points. b, Kaplan–Meier plots for the prediction of the progression from non-referable DR to referable DR. One-sided log-rank test was used for the comparison between the low- and high-risk groups. The P values on internal test set and external validation dataset–1 and 2 are 6.995×10^-24, 6.236×10^-28 and 3.500×10^-9, respectively. c, Prediction of the progression from non-referable DR to referable DR using time-dependent ROC curves. Shaded areas in a and b are 95% CIs. Areas under ROC curves are presented as mean values (lower bound of 95% CI, upper bound of 95% CI). Source data

Extended Data Fig. 5. Internal and external validation of the DeepDR Plus system in the prediction of the progression from non-vision-threatening DR to vision-threatening DR.

a, Integrated Brier score (left) showing overall fit—lower is better and C-index (right) measuring model risk discrimination—higher is better—for various time points. b, Kaplan–Meier plots for the prediction of the progression from non-vision-threatening DR to vision-threatening DR. One-sided log-rank test was used for the comparison between the low- and high-risk groups. The P values on internal test set and external validation dataset–1 and 2 are 8.528×10^-8, 6.647×10^-8 and 7.018×10^-3, respectively. c, Prediction of the progression from non-vision-threatening DR to vision-threatening DR using time-dependent ROC curves. Shaded areas in a and b are 95% CIs. Areas under ROC curves are presented as mean values (lower bound of 95% CI, upper bound of 95% CI). Source data

Extended Data Fig. 6. The real-world study to assess the clinical outcome by integration with DeepDR Plus system.

a, Flowchart of the study and actual DR progression rate among high-risk and low-risk evaluated by the Fundus and Metadata models in IM group and Non-IM group. b, Waterfall plot of predicted time to DR progression of participants in the real-world study by fundus model (DeepDR Plus). The waterfall plot displays the predicted time to DR progression of all participants in the real-world study by the fundus model. The individualized screening interval was set at an annual time point from baseline, which was just the year after the predicted patient-specific time to DR progression by the fundus model. Source data

Extended Data Fig. 7. Illustration of the model structure of DeepDR Plus system.

There are three models (fundus model, metadata model, and combined model) in the DeepDR Plus system, which can support different types of inputs. The fundus model has a fundus feature extractor and a predictor to generate a predicted time to progression of DR and fundus score. The fundus feature extractor is pretrained using Momentum Contrast (MoCo v2) to generate high-level feature vectors, while the predictor estimates the survival time in a fixed-size mixture of Weibull distributions based on the fundus feature vectors to generate the fundus score. The metadata and combined models share the same structure but differ in their inputs compared to the fundus model. The metadata model takes metadata as inputs, while the combined model takes both metadata and fundus score as inputs.