Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 Nov;123(11):2345-2351.
doi: 10.1016/j.ophtha.2016.07.020. Epub 2016 Aug 24.

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis

Jayashree Kalpathy-Cramer  1 , J Peter Campbell  2 , Deniz Erdogmus  3 , Peng Tian  3 , Dharanish Kedarisetti  3 , Chace Moleta  2 , James D Reynolds  4 , Kelly Hutcheson  5 , Michael J Shapiro  6 , Michael X Repka  7 , Philip Ferrone  8 , Kimberly Drenser  9 , Jason Horowitz  10 , Kemal Sonmez  11 , Ryan Swan  11 , Susan Ostmo  2 , Karyn E Jonas  12 , R V Paul Chan  12 , Michael F Chiang  13 ; Imaging and Informatics in Retinopathy of Prematurity Research Consortium
Collaborators, Affiliations

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis

Jayashree Kalpathy-Cramer et al. Ophthalmology. 2016 Nov.

Abstract

Purpose: To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP.

Design: We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases.

Participants: Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units.

Methods: Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system.

Main outcome measures: Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling.

Results: There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86).

Conclusions: Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Graph showing that interexpert agreement using classification ranking (plus vs. preplus vs. normal) was slight to substantial (mean weighted κ, 0.27; range, 0.06–0.63 between pairs of experts).
Figure 2
Figure 2
Scatterplot showing the high correlation between comparison ranks (correlation coefficient, 0.92), suggesting that both of these approaches for attaining consensus ranking of image severity by experts result in consistent severity gradings for retinopathy of prematurity.
Figure 3
Figure 3
Scatterplot showing that the comparison rankings by all 5 graders were highly consistent (mean correlation coefficient, 0.97; range, 0.95–0.98) between each expert and the consensus comparison ranking.
Figure 4
Figure 4
The Imaging and Informatics in Retinopathy of Prematurity (i-ROP) regression score for each of the 9 representative images.
See this image and copyright information in PMC

References

    1. Sommer A, Taylor HR, Ravilla TD, et al. Challenges of ophthalmic care in the developing world. JAMA Ophthalmol. 2014;132:640–644. - PMC - PubMed
    1. Gilbert C, Foster A. Childhood blindness in the context of VISION 2020--the right to sight. Bull World Health Organ. 2001;79:227–232. - PMC - PubMed
    1. Early Treatment for Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121:1684–1694. - PubMed
    1. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of cryotherapy for retinopathy of prematurity. Preliminary results. Arch Ophthalmol. 1988;106:471–479. - PubMed
    1. International Committee for the Classification of Retinopathy of Prematurity. The International Classification of Retinopathy of Prematurity revisited. American Medical Association. 2005;123:991–999. - PubMed

Publication types

MeSH terms