Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training

Abstract

Purpose: To measure the accuracy of image-based retinopathy of prematurity (ROP) diagnosis by pediatric ophthalmology fellows.

Methods: This was a comparative case series of expert versus nonexpert clinicians in image-based ROP diagnosis. An atlas of 804 retinal images was captured from 248 eyes of 67 premature infants with a wide-angle camera (RetCam-II, Clarity Medical Systems, Pleasanton, CA). Images were uploaded to a study website from which an expert pediatric retinal specialist and five pediatric ophthalmology fellows independently provided a diagnosis (no ROP, mild ROP, type 2 ROP, or treatment-requiring ROP) for each eye. Two different retinal specialists experienced in ROP examination served as additional controls. Primary outcome measures were sensitivity and specificity of image-based ROP diagnosis by fellows compared to a reference standard of image-based interpretation by the expert pediatric retinal specialist. Secondary outcome measure was intraphysician reliability.

Results: For detection of mild or worse ROP, the mean (range) sensitivity among the five fellows was 0.850 (0.670-0.962) and specificity was 0.919 (0.832-0.964). For detection of type 2 or worse ROP by fellows, mean (range) sensitivity was 0.527 (0.356-0.709) and specificity was 0.938 (0.777-1.000). For detection of treatment-requiring ROP, mean (range) sensitivity was 0.515 (0.267-0.765) and specificity was 0.949 (0.805-1.00).

Conclusions: Pediatric ophthalmology fellows in this study demonstrated high diagnostic specificity in image-based ROP diagnosis; however, sensitivity was lower, particularly for clinically significant disease.

FIG 1

Mean distribution of ROP (retinopathy of prematurity) diagnoses by pediatric ophthalmology fellows. Of 143 eyes with no ROP, fellows agreed in 127 (89%). Of 62 eyes with mild ROP, fellows agreed in 43 (69%). Of 28 eyes with type 2 ROP, fellows agreed in 6 (20%). Of 15 eyes with treatment-requiring ROP, fellows agreed in 11 (75%).

FIG 2

Example of study images frequently misdiagnosed by fellows. A, B, C, Nasal, posterior, and temporal images diagnosed as type 2 ROP by the expert reference standard and no ROP by 2 of 5 fellows (40%), mild ROP by 1 of 5 fellows (20%), type 2 by 1 of 5 fellows (20%), and treatment-requiring by 1 of 5 of fellows (20%).