Multi-population generalizability of a deep learning-based chest radiograph severity score for COVID-19

Abstract

To tune and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. A published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from 4 test sets, including 3 from the United States (patients hospitalized at an academic medical center (N = 154), patients hospitalized at a community hospital (N = 113), and outpatients (N = 108)) and 1 from Brazil (patients at an academic medical center emergency department (N = 303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson R). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. Tuning the deep learning model with outpatient data showed high model performance in 2 United States hospitalized patient datasets (R = 0.88 and R = 0.90, compared to baseline R = 0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (R = 0.86 and R = 0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. A deep learning model that extracts a COVID-19 severity score on CXRs showed generalizable performance across multiple populations from 2 continents, including outpatients and hospitalized patients.

Figure 1.

Schematic of study design. Previously published Siamese neural network-based model for extracting lung disease severity from CXRs^[7] was tuned using new CXR data and evaluated in 4 test sets.

Figure 2.

Boxplots show variable distributions in patient age (A) and lung disease severity by mRALE score (B) in the different CXR test sets. Boxplots show the median and interquartile range (IQR), where the whiskers extend up to 1.5 x IQR.

Figure 3.

Scatterplots show the correlation between radiologist-determined mRALE score and the deep learning-based PXS score in the Hospital 1 Inpatient Test Set (R = 0.88) (A), Hospital 1 Outpatient Test Set (R = 0.86) (B), Hospital 2 Emergency Test Set (R = 0.85) (C), and Hospital 3 Inpatient Test Set (R = 0.90) (A). Linear regression 95% confidence intervals are shown in each scatterplot.

Figure 4.

Dimensionality reduction using UMAP shows the relationships between CXR data passed through the deep learning-based PXS score model from all 4 test sets (total N = 678), color coded for PXS score (A), mRALE score (B), and test set (C). For the legend in (C), H indicates Hospital. Across the different test sets, a representation of lung disease severity is learned by the PXS score model.