Deep-learning-based enhanced optic-disc photography

Abstract

Optic-disc photography (ODP) has proven to be very useful for optic nerve evaluation in glaucoma. In real clinical practice, however, limited patient cooperation, small pupils, or media opacities can limit the performance of ODP. The purpose of this study was to propose a deep-learning approach for increased resolution and improved legibility of ODP by contrast, color, and brightness compensation. Each high-resolution original ODP was transformed into two counterparts: (1) down-scaled 'low-resolution ODPs', and (2) 'compensated high-resolution ODPs' produced via enhancement of the visibility of the optic disc margin and surrounding retinal vessels using a customized image post-processing algorithm. Then, the differences between these two counterparts were directly learned through a super-resolution generative adversarial network (SR-GAN). Finally, by inputting the high-resolution ODPs into SR-GAN, 4-times-up-scaled and overall-color-and-brightness-transformed 'enhanced ODPs' could be obtained. General ophthalmologists were instructed (1) to assess each ODP's image quality, and (2) to note any abnormal findings, at 1-month intervals. The image quality score for the enhanced ODPs was significantly higher than that for the original ODP, and the overall optic disc hemorrhage (DH)-detection accuracy was significantly higher with the enhanced ODPs. We expect that this novel deep-learning approach will be applied to various types of ophthalmic images.

Fig 1. Principle of enhanced image formation via super-resolution generative adversarial network (SR-GAN).

A modified SR-GAN was used to learn the differences between the low-resolution optic-disc photography (ODP) and the manually compensated high-resolution ODP. By inputting the high-resolution original ODP into the algorithm, an X4 up-scaled and overall contrast-, color- and brightness-transformed ‘enhanced ODP’ could be obtained.

Fig 2. Architecture of generator and discriminator network.

The corresponding kernel size (k), number of feature maps (n) and stride (s) are indicated for each convolutional layer.

Fig 3. Customized image post-processing algorithm for maximized visibility of hemorrhage.

(A) Original ODP. (B) With Selective color tool, red color was replaced by green/blue and (C) yellow was replaced by green/blue. (D) With the Contrast/Brightness tool, the contrast level was improved, and with the Smarten sharpen tool, the degree and range of the sharpness was increased. (E) Finally, a compensated high-resolution ODP could be obtained.

Fig 4. Representative optic-disc photography (ODP) of eye with optic disc hemorrhage (DH).

(A) Magnified image of inferotemporal area in original high-resolution ODP, (B) Magnified image of inferotemporal area in deep-learning-based enhanced ODP. The enhanced ODP improved the color and spatial contrast between the DH and the background retinal color.

Fig 5. Training curve for deep-learning algorithm.

The red line shows the accuracy of the discriminator over the training course, while the blue line represents the accuracy of the generator. As can be seen, after the 800th epoch, the discriminator’s loss decreases and the generator’s loss increases.

Fig 6. Validation results for representative test image sets.

The Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) values were higher in the upper 3 sets, and the obtained optic-disc photography (ODPs) were perceptually similar to the targeted ODPs (A). In the lower image sets, the change in background color caused relatively lower PSNR and SSIM values, even though those images were perceptually convincing (B).

Fig 7. Scatter plot of delta mean opinion score (Δ MOS) against MOS of original optic-disc photography (ODP).

The Δ MOS was calculated as the difference between the original and enhanced ODP scores. Note that the lower the original ODP image quality score, the larger the Δ MOS.

Fig 8. Example of fundus photography transferred from another institution as printed document.

(A) Original document, (B) directly extracted fundus image, (C) magnified image of optic disc in original fundus photography, (D) deep-learning-based enhanced fundus photography, (E) magnified image of optic disc in enhanced fundus photography. The enhanced fundus photography improved the structural details of the optic disc, neuroretinal rim margin and vessel contours.