Automatic classification of heterogeneous slit-illumination images using an ensemble of cost-sensitive convolutional neural networks

Abstract

Background: Lens opacity seriously affects the visual development of infants. Slit-illumination images play an irreplaceable role in lens opacity detection; however, these images exhibited varied phenotypes with severe heterogeneity and complexity, particularly among pediatric cataracts. Therefore, it is urgently needed to explore an effective computer-aided method to automatically diagnose heterogeneous lens opacity and to provide appropriate treatment recommendations in a timely manner.

Methods: We integrated three different deep learning networks and a cost-sensitive method into an ensemble learning architecture, and then proposed an effective model called CCNN-Ensemble [ensemble of cost-sensitive convolutional neural networks (CNNs)] for automatic lens opacity detection. A total of 470 slit-illumination images of pediatric cataracts were used for training and comparison between the CCNN-Ensemble model and conventional methods. Finally, we used two external datasets (132 independent test images and 79 Internet-based images) to further evaluate the model's generalizability and effectiveness.

Results: Experimental results and comparative analyses demonstrated that the proposed method was superior to conventional approaches and provided clinically meaningful performance in terms of three grading indices of lens opacity: area (specificity and sensitivity; 92.00% and 92.31%), density (93.85% and 91.43%) and opacity location (95.25% and 89.29%). Furthermore, the comparable performance on the independent testing dataset and the internet-based images verified the effectiveness and generalizability of the model. Finally, we developed and implemented a website-based automatic diagnosis software for pediatric cataract grading diagnosis in ophthalmology clinics.

Conclusions: The CCNN-Ensemble method demonstrates higher specificity and sensitivity than conventional methods on multi-source datasets. This study provides a practical strategy for heterogeneous lens opacity diagnosis and has the potential to be applied to the analysis of other medical images.

Keywords: Cost-sensitive; deep convolutional neural networks (CNNs); ensemble learning; heterogeneous slit-illumination images; pediatric cataract.

Figure 1

Dataset preparation and performance evaluation of multiple methods. (A) Dataset labelling and preprocessing. Four hundred and seventy training and validation samples and 132 independent test samples were derived from samples provided by the Zhongshan Ophthalmic Center of Sun Yat-sen University; 79 Internet-based samples were collected using the Baidu and Google search engines. Each image was independently graded and labeled by three senior ophthalmologists; subsequently, the images were cropped automatically using twice-applied Canny detection and Hough transformation. (B) Model comparison and evaluation. The training and validation dataset was used to train and evaluate the performances of the different methods and select the best model. Independent testing and Internet-based datasets were also used to evaluate the stability and generalizability of the CCNN-Ensemble method. WT, wavelet transformation; LBP, local binary pattern; SIFT, scale-invariant feature transform; COTE, color and texture features; Adaboost, adaptive boosting ensemble learning; Ave-Ensemble, ensemble learning of three different CNNs (AlexNet, GoogLeNet, and ResNet50) with an averaging technique; Ave-BRS-3ResNet, ensemble learning of three ResNet50 architectures with batch random selection and averaging techniques; CCNN-Ensemble, ensemble learning of cost-sensitive convolutional neural networks.

Figure 2

Framework of the CCNN-Ensemble method. The preprocessed images were input into three parallel deep learning CNNs (AlexNet, GoogLeNet, and ResNet50) with different network structures for feature extraction and classification; a unified ensemble learning of CNNs was then used to improve the recognition rate of the classifier. The cost-sensitive layer was used to adjust the costs of the positive and negative samples in the loss function to address the imbalanced dataset problem. CNN, convolutional neural network; AlexNet, eight-layer Alex CNN; GoogLeNet, 22-layer inception CNN developed by Google researchers; ResNet50, 50-layer residual CNN.

Figure 3

Performance comparisons of the different methods for the three grading indices. Performance comparisons of conventional features, Adaboost ensemble learning, and CCNN-Ensemble methods for the lens opacity area, opacity density, and opacity location, respectively. The sensitivity of Adaboost ensemble learning methods is greatly improved over the conventional feature methods, whereas their specificity indicator is reduced and the accuracy has no significant improvement. The CCNN-Ensemble method outperforms other conventional features and Adaboost ensemble approaches and offers exceptional accuracy, specificity, and sensitivity in terms of three grading indices of lens opacity: area (92.13%, 92.00%, and 92.31%), density (92.77%, 93.85%, and 91.43%) and location (92.76%, 95.25%, and 89.29%). ACC, accuracy; SPE, specificity; SEN, sensitivity; WT, wavelet transformation; LBP, local binary pattern; SIFT, scale-invariant feature transform; COTE, color and texture features; Ada, adaptive boosting ensemble learning; WT-Ada, adaptive boosting ensemble learning with wavelet transformation feature; CCNN-Ensemble, ensemble learning of cost-sensitive convolutional neural networks.

Figure 4

ROC and PR curves for the different methods in opacity area grading. (A) ROC curves and AUC values for the CCNN-Ensemble method and four comparison methods: WT-Ada, SIFT-Ada, LBP-Ada, and COTE-Ada. (B) PR curves for the CCNN-Ensemble method and the four comparison methods. WT, wavelet transformation; LBP, local binary pattern; SIFT, scale-invariant feature transform; COTE, color and texture features; Ada, adaptive boosting ensemble learning; WT-Ada, adaptive boosting ensemble learning with wavelet transformation feature; CCNN-Ensemble, ensemble learning of cost-sensitive convolutional neural networks; ROC, receiver operating characteristic curve; AUC, area under the ROC curve; PR, precision recall curve.

Figure 5

Performance analysis results for the CCNN-Ensemble on two external datasets. (A) The performance comparison, ROC curves, and PR curves of the CCNN-Ensemble method for lens opacity area, density, and location grading on the independent testing dataset. (B) The performance comparison, ROC curves, and PR curves for lens opacity area, density, and location grading on Internet-based dataset. The model performances are satisfactory when applied to the two external datasets, independent test images: area (94.70%, 96.70%, and 90.24%), density (93.18%, 94.23%, and 89.29%) and location (93.18%, 94.00%, and 90.63%); internet-based images: area (89.87%, 89.47%, and 90.00%), density (88.61%, 88.89%, and 88.52%) and location (87.34%, 87.50%, and 87.30%), indicating that the model is universal and effective. ACC, accuracy; SPE, specificity; SEN, sensitivity; ROC, receiver operating characteristic curve; AUC, area under the ROC curve; PR, precision recall curve.

Figure 6

The representative heatmaps of CCNN-Ensemble in opacity area grading using Grad-CAM. (A) The original slit-illumination images. (B,C,D) The visualization heatmaps generated from Alexnet, GoogLeNet, and ResNet50 in the CCNN-Ensemble method. The upper two rows indicate negative samples with limited opacity area, and the lower two rows represent positive samples with extensive opacity area. Grad-CAM, Gradient-weighted Class Activation Mapping.