Uncertainty-inspired open set learning for retinal anomaly identification

Abstract

Failure to recognize samples from the classes unseen during training is a major limitation of artificial intelligence in the real-world implementation for recognition and classification of retinal anomalies. We establish an uncertainty-inspired open set (UIOS) model, which is trained with fundus images of 9 retinal conditions. Besides assessing the probability of each category, UIOS also calculates an uncertainty score to express its confidence. Our UIOS model with thresholding strategy achieves an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set, external target categories (TC)-JSIEC dataset and TC-unseen testing set, respectively, compared to the F1 score of 92.20%, 80.69% and 64.74% by the standard AI model. Furthermore, UIOS correctly predicts high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images. UIOS provides a robust method for real-world screening of retinal anomalies.

Fig. 1. The overview of the uncertainty-inspired open set (UIOS) learning for retinal anomaly classification.

Standard artificial intelligence (AI) and our proposed UIOS AI models were trained with the same dataset with 9 categories of retinal photos. In testing, standard AI model assigns a probability value (p_i) to each of the 9 categories, and the one with the highest probability is output as the diagnosis. Even when the model is tested with a retinal image with disease outside the training set, the model still outputs one from the 9 categories, which may lead to misdiagnosis. In contrast, UIOS outputs an uncertainty score ($\mu$) besides the probability (p_i) for the 9 categories. When the model is fed with an image with obvious features of retinal disease in the 9 categories, the uncertainty-based classifier will output a prediction result with a low uncertainty score below the threshold $\theta$ to indicate that the diagnosis result is reliable. Conversely, when the input data contains ambiguous features or is an anomaly outside of training categories, the model will assign a high uncertainty score above threshold $\theta$ to explicitly indicate that the prediction result is unreliable and requires a double-check from their ophthalmologist to avoid misdiagnosis.

Fig. 2. The receiver operating characteristic (ROC) curves of the standard AI model, our UIOS, and UIOS+thresholding in internal and two external testing datasets.

Source data are provided as a Source data file.

Fig. 3. Uncertainty density distribution for different datasets.

Different colored solid lines indicate different test datasets for target categories of retinal diseases, while different colored dashed lines indicate different out of distribution datasets. θ threshold theta. Source data are provided as a Source data file.

Fig. 4. Four samples of fundus images detected with the standard AI model and our UIOS model.

a, b Two samples with correct diagnostic results from both the standard AI model and our UIOS model. c, d Two samples with incorrect diagnostic results from the standard AI model and our UIOS model. Unlike the standard AI model, which directly takes the fundus disease category with the highest probability score as the final diagnosis result, our UIOS will not only give the probability scores but also provide the corresponding uncertainty score to reflect the reliability of the prediction result. If the uncertainty score is less than the threshold theta, indicating the model prediction is reliable; Conversely, if the uncertainty score is greater than the threshold theta, which represents that the result is unreliable, and manual double-checking is required to avoid possible misdiagnosis problems. US uncertainty score, θ threshold theta. Source data are provided as a Source data file.

Fig. 5. Testing results of OOD samples that were not included in the training category.

Besides assigning a probability to OOD samples as the standard AI model, our UIOS model also assigns a high uncertainty score to indicate that the final decision is unreliable and needs a double-check. US uncertainty score, θ threshold theta. Source data are provided as a Source data file. a from NTC dataset, b also from NTC dataset, c from low-quality dataset, d an OCT image from RETOUCH dataset, and e an OCTA image from OCTA dataset.