A Multimodal Imaging-Based Deep Learning Model for Detecting Treatment-Requiring Retinal Vascular Diseases: Model Development and Validation Study

Abstract

Background: Retinal vascular diseases, including diabetic macular edema (DME), neovascular age-related macular degeneration (nAMD), myopic choroidal neovascularization (mCNV), and branch and central retinal vein occlusion (BRVO/CRVO), are considered vision-threatening eye diseases. However, accurate diagnosis depends on multimodal imaging and the expertise of retinal ophthalmologists.

Objective: The aim of this study was to develop a deep learning model to detect treatment-requiring retinal vascular diseases using multimodal imaging.

Methods: This retrospective study enrolled participants with multimodal ophthalmic imaging data from 3 hospitals in Taiwan from 2013 to 2019. Eye-related images were used, including those obtained through retinal fundus photography, optical coherence tomography (OCT), and fluorescein angiography with or without indocyanine green angiography (FA/ICGA). A deep learning model was constructed for detecting DME, nAMD, mCNV, BRVO, and CRVO and identifying treatment-requiring diseases. Model performance was evaluated and is presented as the area under the curve (AUC) for each receiver operating characteristic curve.

Results: A total of 2992 eyes of 2185 patients were studied, with 239, 1209, 1008, 211, 189, and 136 eyes in the control, DME, nAMD, mCNV, BRVO, and CRVO groups, respectively. Among them, 1898 eyes required treatment. The eyes were divided into training, validation, and testing groups in a 5:1:1 ratio. In total, 5117 retinal fundus photos, 9316 OCT images, and 20,922 FA/ICGA images were used. The AUCs for detecting mCNV, DME, nAMD, BRVO, and CRVO were 0.996, 0.995, 0.990, 0.959, and 0.988, respectively. The AUC for detecting treatment-requiring diseases was 0.969. From the heat maps, we observed that the model could identify retinal vascular diseases.

Conclusions: Our study developed a deep learning model to detect retinal diseases using multimodal ophthalmic imaging. Furthermore, the model demonstrated good performance in detecting treatment-requiring retinal diseases.

Keywords: deep learning; detection; eye; imaging; machine learning; model; multimodal imaging; retinal; retinal vascular diseases; treatment; treatment requirement; vascular.

Figure 1

Flowchart of multimodal image management and processing. OCT: optical coherence tomography; FA/ICGA: fluorescein angiography with or without indocyanine green angiography.

Figure 2

Architecture of the deep learning prediction model. CNN: convolutional neural network; FCL: fully connected layer; GAP: global average pooling.

Figure 3

Receiver operating characteristic curves of the model performance for (A) predicting different retinal vascular diseases and (B) identifying treatment-requiring diseases. AUC: area under the curve; BRVO: branch retinal vein occlusion; CRVO: central retinal vein occlusion; DME: diabetic macular edema; mCNV: myopic choroidal neovascularization; nAMD: neovascular age-related macular degeneration.

Figure 4

Confusion matrix demonstrating the performance of the prediction model in different retinal vascular diseases. BRVO: branch retinal vein occlusion; CRVO: central retinal vein occlusion; DME: diabetic macular edema; mCNV: myopic choroidal neovascularization; nAMD: neovascular age-related macular degeneration.

Figure 5

Sample heat maps generated by the prediction model in a true-positive patient with (A) treatment-requiring branch retinal vein occlusion, (B) treatment-requiring diabetic macular edema, and (C) non–treatment-requiring age-related macular degeneration.

Figure 6

Sample heat maps for false prediction of the model: (A) false prediction of treatment-requiring diabetic macular edema (DME) in a patient with coexisting DME and central retinal vein occlusion (CRVO); (B) false prediction of treatment-requiring age-related macular degeneration (AMD) in a patient with central serous chorioretinopathy; (C) false prediction of treatment-requiring DME in a patient with epiretinal membrane, lamellar macular hole, and diabetic retinopathy; (D) false prediction of treatment-requiring DME in a patient with advanced CRVO.