A cascade eye diseases screening system with interpretability and expandability in ultra-wide field fundus images: A multicentre diagnostic accuracy study

Abstract

Background: Clinical application of artificial intelligence is limited due to the lack of interpretability and expandability in complex clinical settings. We aimed to develop an eye diseases screening system with improved interpretability and expandability based on a lesion-level dissection and tested the clinical expandability and auxiliary ability of the system.

Methods: The four-hierarchical interpretable eye diseases screening system (IEDSS) based on a novel structural pattern named lesion atlas was developed to identify 30 eye diseases and conditions using a total of 32,026 ultra-wide field images collected from the Second Affiliated Hospital of Zhejiang University, School of Medicine (SAHZU), the First Affiliated Hospital of University of Science and Technology of China (FAHUSTC), and the Affiliated People's Hospital of Ningbo University (APHNU) in China between November 1, 2016 to February 28, 2022. The performance of IEDSS was compared with ophthalmologists and classic models trained with image-level labels. We further evaluated IEDSS in two external datasets, and tested it in a real-world scenario and an extended dataset with new phenotypes beyond the training categories. The accuracy (ACC), F1 score and confusion matrix were calculated to assess the performance of IEDSS.

Findings: IEDSS reached average ACCs (aACC) of 0·9781 (95%CI 0·9739-0·9824), 0·9660 (95%CI 0·9591-0·9730) and 0·9709 (95%CI 0·9655-0·9763), frequency-weighted average F1 scores of 0·9042 (95%CI 0·8957-0·9127), 0·8837 (95%CI 0·8714-0·8960) and 0·8874 (95%CI 0·8772-0·8972) in datasets of SAHZU, APHNU and FAHUSTC, respectively. IEDSS reached a higher aACC (0·9781, 95%CI 0·9739-0·9824) compared with a multi-class image-level model (0·9398, 95%CI 0·9329-0·9467), a classic multi-label image-level model (0·9278, 95%CI 0·9189-0·9366), a novel multi-label image-level model (0·9241, 95%CI 0·9151-0·9331) and a lesion-level model without Adaboost (0·9381, 95%CI 0·9299-0·9463). In the real-world scenario, the aACC of IEDSS (0·9872, 95%CI 0·9828-0·9915) was higher than that of the senior ophthalmologist (SO) (0·9413, 95%CI 0·9321-0·9504, p = 0·000) and the junior ophthalmologist (JO) (0·8846, 95%CI 0·8722-0·8971, p = 0·000). IEDSS remained strong performance (ACC = 0·8560, 95%CI 0·8252-0·8868) compared with JO (ACC = 0·784, 95%CI 0·7479-0·8201, p= 0·003) and SO (ACC = 0·8500, 95%CI 0·8187-0·8813, p = 0·789) in the extended dataset.

Interpretation: IEDSS showed excellent and stable performance in identifying common eye conditions and conditions beyond the training categories. The transparency and expandability of IEDSS could tremendously increase the clinical application range and the practical clinical value of it. It would enhance the efficiency and reliability of clinical practice, especially in remote areas with a lack of experienced specialists.

Funding: National Natural Science Foundation Regional Innovation and Development Joint Fund (U20A20386), Key research and development program of Zhejiang Province (2019C03020), Clinical Medical Research Centre for Eye Diseases of Zhejiang Province (2021E50007).

Keywords: Artificial intelligence; Enpandability; Eye diseases screening system; Interpretability; Ultra-wide field fundus image.

Figure 1

The clinical logical relation of lesion atlas. The lesion atlas integrated the pathological and anatomical information (a). The anatomical location was determined by dividing the retina into 21 regions as the prototype (b) and the overlaid image (c) displayed. DR, diabetic retinopathy; AMD, age-related macular degeneration; CRVO, central retinal vein occlusion; BRVO, branch retinal vein occlusion; CSCR, central serous chorioretinopathy; RT, retinal tears; EMM, epimacular membrane; MH, macular Hole; MHE, macular hemorrhage; ODE, optic disc edema; OA, optic atrophy; OPN, optic perineuritis; SG, suspected glaucoma; CNV, choroidal neovascularization; HE, hard exudates; CWS, cotton wool spots; MA, microaneurysm; IRH, intraretinal hemorrhage; SRH, subretinal hemorrhage; PRH, preretinal hemorrhage; SRD, serous retinal detachment; ERM, epiretinal membrane; ICDR increased cup-to-disc ratio; ODP, optic disc pallor.

Figure 2

The workflow of overall study. Images were collected from three clinical centres and filtered by quality assessment. The preliminary screening module was constructed to provide automatic referral for patients with retinal detachment and vitreous hemorrhage, with the rest followed by preprocessing and lesion-level annotation. The dominant screening module consisted of three parts: (1) the location algorithm locating the optic disc region and macula region, and dividing the retina into 21 regions; (2) the lesion atlas mapping algorithm extracting the pathological and anatomical information of lesions automatically; (3) the mass screening algorithm integrating extracted features and making a final decision. The performance of the system was compared with image-level models and evaluated in a multicentric scenario, a real-world scenario and an expanded scenario in multimorbidity dataset.

Figure 3

Overview pipeline of Interpretable eye diseases screening system (IEDSS). There were two paths in the lesion atlas mapping module. In the upper path, geometric regions were generated after ONH segmentation and macula localization. In the lower path, multi-scale patches were cropped from a CLAHE-ed UWF image, and sent into the CAFPN. Results of the lesion detection were output from the CAPFN, and summarized into the lesion atlas with geometric regions (a). The lesion geostatisitcs showed the lesion distribution in the lesion atlas and was flatten to get the DFV. The input feature was generated by DFV dimensionality reduction through PCA and input into the AdaBoost-SVM to get multi-label probabilities in the mass screening system (b). In the training process of the AdaBoost-SVM, Weak classifiers were trained iteratively, and assembled to the strong classifier (c). ONH, optic nerve head; HE, hard exudates; MA, microaneurysm; IRH, intraretinal hemorrhage; CAFPN, channel-attention feature pyramid network; CLAHE, contrast limited adaptive histogram equalization; AdaBoost-SVM, AdaBoost-support vector machine; DFV, discrimination feature vectors; PCA, principal components analysis; ROI, region of interest.

Figure 4

The lesion geostatistics heatmap generated by lesion atlas illustrating the type, number and distribution of lesions in different diseases. CNV, choroidal neovascularization; HE, hard exudates; CWS, cotton wool spots; MA, microaneurysm; IRH, intraretinal hemorrhage; SRH, subretinal hemorrhage; PRH, preretinal hemorrhage; SRD, serous retinal detachment; ERM, epiretinal membrane; MH, macular holes; RT, retinal tears; ODE, optic disc edema; ODP, optic disc pallor; ICDR, increased cup-to-disc ratio.

Figure 5

The t-Distributed stochastic neighbor embedding visualizations of discrimination feature vectors in the screening model for the classification of 11 diseases (a). The map preserved the topologic features, and adjacent coordinates meant they shared similar features in the original feature space. A selection of adjacent cases showing similar clinical features (b-e). DR, diabetic retinopathy; CSCR, central serous chorioretinopathy; AMD, age-related macular degeneration; MH, macular Hole; EMM, epimacular membrane; CRVO, central retinal vein occlusion; BRVO, branch retinal vein occlusion; RT, retinal tears; ODE, optic disc edema; OA, optic atrophy; SG, suspected glaucoma.

Figure 6

The diagnosis of 30 examples of multimorbidity by the IEDSS and two ophthalmologists with (without) assistance of Interpretable eye diseases screening system (IEDSS). Correct diagnosis was defined as successfully identifying all types of diseases. The wrong samples were analysed and summarized the reasons into the following three types: 1. misidentification, which was defined as the lesion was correctly detected but the disease was misdiagnosed, 2. undetected lesions, which was defined as the missed detection of lesions, 3. misdetected lesions, including false positives and wrongly categorized lesions. DR, diabetic retinopathy; CSCR, central serous chorioretinopathy; AMD, age-related macular degeneration; MH, macular Hole; EMM, epimacular membrane; CRVO, central retinal vein occlusion; BRVO, branch retinal vein occlusion; ODE, optic disc edema; OA, optic atrophy; SG, suspected glaucoma.