Use of an Artificial Intelligence-Generated Vascular Severity Score Improved Plus Disease Diagnosis in Retinopathy of Prematurity

Abstract

Purpose: To evaluate whether providing clinicians with an artificial intelligence (AI)-based vascular severity score (VSS) improves consistency in the diagnosis of plus disease in retinopathy of prematurity (ROP).

Design: Multireader diagnostic accuracy imaging study.

Participants: Eleven ROP experts, 9 of whom had been in practice for 10 years or more.

Methods: RetCam (Natus Medical Incorporated) fundus images were obtained from premature infants during routine ROP screening as part of the Imaging and Informatics in ROP study between January 2012 and July 2020. From all available examinations, a subset of 150 eye examinations from 110 infants were selected for grading. An AI-based VSS was assigned to each set of images using the i-ROP DL system (Siloam Vision). The clinicians were asked to diagnose plus disease for each examination and to assign an estimated VSS (range, 1-9) at baseline, and then again 1 month later with AI-based VSS assistance. A reference standard diagnosis (RSD) was assigned to each eye examination from the Imaging and Informatics in ROP study based on 3 masked expert labels and the ophthalmoscopic diagnosis.

Main outcome measures: Mean linearly weighted κ value for plus disease diagnosis compared with RSD. Area under the receiver operating characteristic curve (AUC) and area under the precision-recall curve (AUPR) for labels 1 through 9 compared with RSD for plus disease.

Results: Expert agreement improved significantly, from substantial (κ value, 0.69 [0.59, 0.75]) to near perfect (κ value, 0.81 [0.71, 0.86]), when AI-based VSS was integrated. Additionally, a significant improvement in plus disease discrimination was achieved as measured by mean AUC (from 0.94 [95% confidence interval (CI), 0.92-0.96] to 0.98 [95% CI, 0.96-0.99]; difference, 0.04 [95% CI, 0.01-0.06]) and AUPR (from 0.86 [95% CI, 0.81-0.90] to 0.95 [95% CI, 0.91-0.97]; difference, 0.09 [95% CI, 0.03-0.14]).

Conclusions: Providing ROP clinicians with an AI-based measurement of vascular severity in ROP was associated with both improved plus disease diagnosis and improved continuous severity labeling as compared with an RSD for plus disease. If implemented in practice, AI-based VSS could reduce interobserver variability and could standardize treatment for infants with ROP.

Financial disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Keywords: Artificial intelligence; Assistive diagnosis; Retinopathy of prematurity.

Figure 1:. Online image grading platform.

Experts logged in to grade image sets (5 images/set on a carousel) for plus disease and to assign a vascular severity score (VSS). Initially, the “predicted” AI-derived VSS was not shown to graders. One month later, the same experts were tasked with grading the same images, however the predicted VSS was displayed.

Figure 2.. Visual representation of plus disease labels for each reader.

A) Without AI-VSS, there was significant variability in both pre-plus and plus diagnosis by expert readers. B) With AI-VSS, there was notable improvement for plus disease, but persistent variability in the frequency of diagnosis of pre-plus disease label (i.e., Readers 1 and 2 used this label significantly less than Readers 3–11).

Figure 3:. Expert agreement of VSS with and without the assistance of AI-VSS.

Each image is plotted twice, once with (cyan) and without (red) AI-VSS assistance, using the average expert VSS and the proportion of experts that agreed with said VSS within ± 1. Quadratic lines ± standard error for with and without AI-VSS were also fit.