Plus disease in retinopathy of prematurity: pilot study of computer-based and expert diagnosis

Abstract

Purpose: To measure accuracy of plus disease diagnosis by recognized experts in retinopathy of prematurity (ROP), and to conduct a pilot study examining performance of a computer-based image analysis system, Retinal Image multiScale Analysis (RISA).

Methods: Twenty-two ROP experts independently interpreted a set of 34 wide-angle retinal images for presence of plus disease. A reference standard diagnosis based on expert consensus was defined for each image. Images were analyzed by the computer-based system using individual and linear combinations of system parameters for arterioles and venules: integrated curvature (IC), diameter, and tortuosity index (TI). Sensitivity, specificity, and receiver operating characteristic areas under the curve (AUC) for plus disease diagnosis compared with the reference standard were determined for each expert, as well as for the computer-based system.

Results: Expert sensitivity ranged from 0.308 to 1.000, specificity ranged from 0.571 to 1.000, and AUC ranged from 0.784 to 1.000. Among individual computer system parameters, venular IC had highest AUC (0.853). Among all computer system parameters, the linear combination of arteriolar IC, arteriolar TI, venular IC, venular diameter, and venular TI had highest AUC (0.967), which was greater than that of 18 (81.8%) of 22 experts.

Conclusions: Accuracy of ROP experts for plus disease diagnosis is imperfect. A computer-based image analysis system has potential to diagnose plus disease with high accuracy. Further research involving RISA system parameter cut-off values from this study are required to fully validate performance of this computer-based system compared with that of human experts.

Figure 1. Parameters of computer-based Retinal Image multiScale Analysis (RISA) system

(A) Tortuosity index (TI) is defined as the length of a vessel divided by a line segment connecting the end points. (B) Integrated curvature (IC) is computed by forming vectors along the vessel, calculating cumulative sum of angles θ between vectors, and normalizing by vessel length. If a vessel is perfectly straight, then IC has a minimum value of zero and TI has a minimum value of one. (C) TI vs. IC. Departures from linearity, such as the vessel depicted on right, are captured both by IC and TI. However, multiple changes in vessel depicted on left are better captured by IC. TI is similar for the two shapes (5% difference), whereas IC is higher for the vessel depicted on left (50% difference).

Figure 2. Box plots of computer-based system parameter values in images with “not plus” compared to “plus,” based on reference standard diagnosis

Individual parameters of (A) integrated curvature (IC), (B) diameter and (C) tortuosity index (TI) for arterioles and venules; as well as combined parameters of (D) Linear combination I (arteriolar IC, venular IC, and venular diameter), and (E) Linear Combination II (arteriolar IC, arteriolar TI, venular IC, venular diameter, and venular TI) are displayed. Boxes represent the 25^th, 50^th, and 75^th percentile values of parameters. Whiskers represent 10^th and 90^th percentile values. Asterisks represent outlier values. P-values indicated are for Mann-Whitney tests between “not plus” and “plus” groups.

Figure 3. Sensitivity and specificity of individual computer system parameters and linear combinations of parameters for plus disease diagnosis, compared to the reference standard of majority vote among 22 recognized ROP experts

Curves are displayed as a function of parameter cutoff criteria for detection of plus disease: (A) arteriolar integrated curvature (IC), (B) arteriolar diameter, (C) arteriolar tortuosity index (TI), (D) venular IC, (E) venular diameter, (F) venular TI, (G) Linear Combination I (arteriolar IC, venular IC and venular diameter) and (H) Linear Combination II (arteriolar IC, arteriolar TI, venular IC, venular diameter, and venular TI).