Variability in Plus Disease Diagnosis using Single and Serial Images

Abstract

Purpose: To assess changes in retinopathy of prematurity (ROP) diagnosis in single and serial retinal images.

Design: Cohort study.

Participants: Cases of ROP recruited from the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) consortium evaluated by 7 graders.

Methods: Seven ophthalmologists reviewed both single and 3 consecutive serial retinal images from 15 cases with ROP, and severity was assigned as plus, preplus, or none. Imaging data were acquired during routine ROP screening from 2011 to 2015, and a reference standard diagnosis was established for each image. A secondary analysis was performed using the i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1 to 9, with 9 being the most severe disease. This score has been previously demonstrated to correlate with the International Classification of ROP. Mean plus disease severity was calculated by averaging 14 labels per image in serial and single images to decrease noise.

Main outcome measures: Grading severity of ROP as defined by plus, preplus, or no ROP.

Results: Assessment of serial retinal images changed the grading severity for > 50% of the graders, although there was wide variability. Cohen's kappa ranged from 0.29 to 1.0, which showed a wide range of agreement from slight to perfect by each grader. Changes in the grading of serial retinal images were noted more commonly in cases of preplus disease. The mean severity in cases with a diagnosis of plus disease and no disease did not change between single and serial images. The ROP VSS demonstrated good correlation with the range of expert classifications of plus disease and overall agreement with the mode class (P = 0.001). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient, 0.89). The more aggressive graders tended to be influenced by serial images to increase the severity of their grading. The VSS also demonstrated agreement with disease progression across serial images, which progressed to preplus and plus disease.

Conclusions: Clinicians demonstrated variability in ROP diagnosis when presented with both single and serial images. The use of deep learning as a quantitative assessment of plus disease has the potential to standardize ROP diagnosis and treatment.

Keywords: ROP imaging; ROP progression; ROP screening; Retinopathy of prematurity; telemedicine.

Figure 1.

The confusion matrices and weighted kappa values for the assessment of single images versus the same image presented at the end of a series of images (serial), for each grader.

Figure 2.

The mean and mode severity of single and serial images. Using a numeric scale which corresponds to the color spectrum (1 = no plus = green, 5 = preplus, and 9 = plus = red), the above table demonstrates the mean (left) and mode (right) values for each image for the group of experts presented with the single image and serial image set. Qualitatively, the mean severity was higher when the images were presented in series.

Figure 3.

Compared with the standard single image grading, this figure presents the changes in diagnosis across all graders after viewing serial images, organized from least to most aggressive graders (left to right). Graders are presented in the order of least (A) to most aggressive (G) grading. Grayscale boxes indicate images with no change in single versus serial images (white = no retinopathy of prematurity [ROP], light gray = preplus, dark gray = plus). The red boxes indicate a change in diagnosis toward more severe disease with serial image viewing (light red = no ROP to preplus, dark red = preplus to plus). The blue boxes indicate a change in diagnosis toward less severe disease (dark blue = plus to preplus, light blue = preplus to no disease).

Figure 4.

Two sets of serial images of cases with a reference standard diagnosis of preplus disease. For single images, graders viewed only the last image in the series. In set (A), the graders showed diagnostic changes toward more severe disease with serial images. In set (B), the graders showed clinical diagnostic changes toward less severe disease.

Figure 5.

A, Vascular severity scores demonstrated good overall agreement with the mode class as assigned to no plus, preplus, and plus disease (P = 0.001). B, The retinopathy of prematurity (ROP) vascular severity score correlated with the range of disease severity, as determined by averaging the clinician responses of disease severity (no plus disease = 1, preplus = 5, plus disease = 9) with a correlation coefficient of 0.8.