Federated Learning for Multicenter Collaboration in Ophthalmology: Implications for Clinical Diagnosis and Disease Epidemiology

Abstract

Objective: To utilize a deep learning (DL) model trained via federated learning (FL), a method of collaborative training without sharing patient data, to delineate institutional differences in clinician diagnostic paradigms and disease epidemiology in retinopathy of prematurity (ROP).

Design: Evaluation of a diagnostic test or technology.

Subjects and controls: We included 5245 patients with wide-angle retinal imaging from the neonatal intensive care units of 7 institutions as part of the Imaging and Informatics in ROP study. Images were labeled with the clinical diagnoses of plus disease (plus, preplus, no plus), which were documented in the chart, and a reference standard diagnosis was determined by 3 image-based ROP graders and the clinical diagnosis.

Methods: Demographics (birth weight, gestational age) and clinical diagnoses for all eye examinations were recorded from each institution. Using an FL approach, a DL model for plus disease classification was trained using only the clinical labels. The 3 class probabilities were then converted into a vascular severity score (VSS) for each eye examination, as well as an "institutional VSS," in which the average of the VSS values assigned to patients' higher severity ("worse") eyes at each examination was calculated for each institution.

Main outcome measures: We compared demographics, clinical diagnoses of plus disease, and institutional VSSs between institutions using the McNemar-Bowker test, 2-proportion Z test, and 1-way analysis of variance with post hoc analysis by the Tukey-Kramer test. Single regression analysis was performed to explore the relationship between demographics and VSSs.

Results: We found that the proportion of patients diagnosed with preplus disease varied significantly between institutions (P < 0.001). Using the DL-derived VSS trained on the data from all institutions using FL, we observed differences in the institutional VSS and the level of vascular severity diagnosed as no plus (P < 0.001) across institutions. A significant, inverse relationship between the institutional VSS and mean gestational age was found (P = 0.049, adjusted R² = 0.49).

Conclusions: A DL-derived ROP VSS developed without sharing data between institutions using FL identified differences in the clinical diagnoses of plus disease and overall levels of ROP severity between institutions. Federated learning may represent a method to standardize clinical diagnoses and provide objective measurements of disease for image-based diseases.

Keywords: Deep learning; Epidemiology; Federated learning; Retinopathy of prematurity.

Figure 1:. Distribution of vascular severity score by clinical diagnosis by clinical site

Box and whisker plot illustrating median (central line), lower and upper quartiles (vertical box bounds) and range (vertical whiskers) of “worse” eye vascular severity scores (VSS) for all exams at each participating site for no plus (A), pre-plus (B) and plus (C) disease. Outliers are represented by scattered dots. Comparison of VSS distribution between institutions for each severity level revealed a significant difference in no plus grading (One way ANOVA: p<0.001, F = 10.18). During pairwise comparison of institutions, 9 / 21 (42.9%) significant differences were found for “no plus” diagnosis.

Figure 2:. Variability in plus disease severity grading

Representative fundus photo of disagreement between clinical diagnosis, reference standard diagnosis (RSD) and vascular severity score (VSS). This image was diagnosed as “no plus” by the clinician at the bedside (Site F), but had a RSD of “pre-plus”, and a VSS of 8.5.

Figure 3:. Distribution of vascular severity scores per site

These plots illustrate the distribution of “worse” eye vascular severity scores (VSS) from each eye exam at each site. For each site, VSS scores are ranked from lowest to highest, and color coded for each of three clinical diagnoses of plus disease. The plots reveal differences in overall ROP severity (e.g. Site B had comparably few exams, and no plus cases), as well as differences in the clinical diagnosis of pre-plus and plus disease (e.g. Site F had very few cases of pre-plus diagnosed even though they had similar VSS distribution to other sites).

Figure 4:. Relationship between institutional vascular severity score and mean population birth weight and gestational age

Simple linear regression revealed a significant, inverse relationship between the institutional vascular severity score (VSS) and (A) gestational age (p=0.049), adjusted R²=0.49), but not (B) birth weight (p=0.10, adjusted R²=0.34).