Estimation of best corrected visual acuity based on deep neural network

Abstract

In this study, we investigated a convolutional neural network (CNN)-based framework for the estimation of the best-corrected visual acuity (BCVA) from fundus images. First, we collected 53,318 fundus photographs from the Gyeongsang National University Changwon Hospital, where each fundus photograph is categorized into 11 levels by retrospective medical chart review. Then, we designed 4 BCVA estimation schemes using transfer learning with pre-trained ResNet-18 and EfficientNet-B0 models where both regression and classification-based prediction are taken into account. According to the results of the study, the predicted BCVA by CNN-based schemes is close to the actual value such that 94.37% of prediction accuracy can be achieved when 3 levels of difference can be tolerated during prediction. The mean squared error and [Formula: see text] score were measured as 0.028 and 0.654, respectively. These results indicate that the BCVA can be predicted accurately for extreme cases, i.e., the level of BCVA is close to either 0.0 or 1.0. Moreover, using the Guided Grad-CAM, we confirmed that the macula and the blood vessel surrounding the macula are mainly utilized in the prediction of BCVA, which validates the rationality of the CNN-based BCVA estimation schemes since the same area is also exploited during the retrospective medical chart review. Finally, we applied the t-distributed stochastic neighbor embedding to examine the characteristics of CNN-based BCVA estimation schemes. The developed BCVA estimation schemes can be employed to obtain the objective measurement of BVCA as well as the medical screening of people with poor access to medical care through smartphone-based fundus imaging.

Figure 1

Confusion matrix of considered schemes.

Figure 2

Histograms of considered schemes for each BCVA level. The last subgraph corresponds to the histogram of all prediction results and actual BCVA levels.

Figure 3

Class activation visualization of the considered schemes with fundus image. For each sub-figure, the first, second, third, and fourth image correspond to the original fundus image, Grad CAM results, Guided Back-propagation result, and Guided Grad CAM results, which combine Grad CAM and Guided Back-propagation, respectively.

Figure 4

Wrong prediction result of the considered scheme when actual BCVA areis 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0. The actual and predicted BCVA is indicated at the top of the figure.

Figure 5

T-SNE result of considered schemes.

Figure 6

The procedure considered the BCVA estimation scheme. The fundus image was resized to $224\times 224\times 3$, and the random flipping and rotating are applied as a means of data augmentation. The modified fundus images were then fed into pre-trained ResNet-18 and EfficientNet-B0, whose outputs were fed into a DNN structure composed of a fully connected layer, batch normalization, dropout, and ReLU. The softmax function is used as a last layer for classification schemes (i.e., Res-Cla and Eff-Cla) whereas the regression schemes (i.e., Res-Reg and Eff-Reg) employ the sigmoid function instead. The parameters of the considered DNN structure are updated using cross-entropy loss and MSE loss using the level of BCVA as a label for classification schemes and regression schemes, respectively.