Deep learning based detection of cone photoreceptors with multimodal adaptive optics scanning light ophthalmoscope images of achromatopsia

Abstract

Fast and reliable quantification of cone photoreceptors is a bottleneck in the clinical utilization of adaptive optics scanning light ophthalmoscope (AOSLO) systems for the study, diagnosis, and prognosis of retinal diseases. To-date, manual grading has been the sole reliable source of AOSLO quantification, as no automatic method has been reliably utilized for cone detection in real-world low-quality images of diseased retina. We present a novel deep learning based approach that combines information from both the confocal and non-confocal split detector AOSLO modalities to detect cones in subjects with achromatopsia. Our dual-mode deep learning based approach outperforms the state-of-the-art automated techniques and is on a par with human grading.

Keywords: (100.2960) Image analysis; (100.4996) Pattern recognition, neural networks; (110.1080) Active or adaptive optics; (170.4470) Ophthalmology.

Fig. 1

Dual-mode AOSLO cone imaging in ACHM subjects. (a) Split detector AOSLO image near the fovea of an ACHM subject. (b) Simultaneously captured confocal AOSLO image from the same location as (a). (c) Split detector AOSLO image at 12° from the fovea in another subject with ACHM. (d) Simultaneously captured confocal AOSLO image from the same location as (c). Orange arrows point to ambiguous locations in the split detector image (c) that can be seen to be cones based on the dark circles in the confocal image (d). Scale bars: 20 μm.

Fig. 2

Schematic of the dual-mode CNN AOSLO cone detection algorithm.

Fig. 3

Extraction of labeled patches from AOSLO image pairs. (a) Cropped split detector AOSLO image. (b) Simultaneously captured cropped confocal AOSLO image from the same location. Voronoi diagram overlain in cyan, manually marked cones are shown in green, and randomly generated locations along Voronoi edges are shown in yellow. (c) Example cone patch pair from position shown in purple in (a) and (b). (d) Example non-cone patch pair from position shown in red in (a) and (b).

Fig. 4

Proposed late fusion dual-mode CNN (LF-DM-CNN) architecture, which consists of the following layers: convolutional (Conv(N,F) where N is the number of kernels, and F is the kernel size in the first two dimensions), fully connected (FC(X) where X is the number of output nodes) batch normalization (BatchNorm), max pooling (MaxPool), average pooling (AvePool), ReLu, concatenation, and soft-max.

Fig. 5

Filter weights from the first convolutional layer in the LF-DM-CNN for the (a) split detector and (b) confocal paths.

Fig. 6

Detection of cones in split detector and confocal AOSLO image pairs. (a) Split detector AOSLO image. (b) Simultaneously captured confocal AOSLO image from the same location. (c) Probability maps generated from (a) and (b) using the trained LF-DM-CNN. (d) Extended maxima of (c). (e-f) Detected cones marked in green on the split detector image shown in (a) and on the confocal image shown in (b).

Fig. 7

Performance of the automated cone detection algorithms on an ACHM image pair. (a) Split detector AOSLO image. (b) Simultaneously captured confocal AOSLO image from the same location. (c-i) Comparison to the first manual markings (with Dice’s coefficients) for (c) the second manual markings (0.915), (d) Bergeles et al. [48] (0.667), (e) C-CNN [46] (0.178), (f) SD-CNN [46] (0.800), (g) C-CNN-ACHM (0.835), (h) SD-CNN-ACHM (0.907), and (i) our proposed method using the LF-DM-CNN network (0.932). Green points denote true positives, cyan denotes false negatives, and red denotes false positives.

Fig. 8

Comparison of our dual-mode method to the single-mode Cunefare et al. [46] method with the SD-CNN-ACHM. Split detector AOSLO images from different subjects with ACHM are shown in the top row, and the corresponding simultaneously captured confocal AOSLO images are shown in the row second from the top. Comparisons to the first manual markings for the single-mode SD-CNN-ACHM are shown in the second row from the bottom, and our method using the dual-mode LF-DM-CNN are shown in the bottom row. Green points denote true positives, cyan denotes false negatives, and red denotes false positives. Orange arrows point to ambiguous locations in the split detector images. Dice’s coefficients for the SD-CNN-ACHM are 0.914 in (a), 0.867 in (b), and 0.815 in (c). Dice’s coefficients for the LF-DM-CNN are 0.986 in (a), 0.929 in (b), and 0.897 in (c).