Anatomy-specific Progression Classification in Chest Radiographs via Weakly Supervised Learning

Abstract

Purpose To develop a machine learning approach for classifying disease progression in chest radiographs using weak labels automatically derived from radiology reports. Materials and Methods In this retrospective study, a twin neural network was developed to classify anatomy-specific disease progression into four categories: improved, unchanged, worsened, and new. A two-step weakly supervised learning approach was employed, pretraining the model on 243 008 frontal chest radiographs from 63 877 patients (mean age, 51.7 years ± 17.0 [SD]; 34 813 [55%] female) included in the MIMIC-CXR database and fine-tuning it on the subset with progression labels derived from consecutive studies. Model performance was evaluated for six pathologic observations on test datasets of unseen patients from the MIMIC-CXR database. Area under the receiver operating characteristic (AUC) analysis was used to evaluate classification performance. The algorithm is also capable of generating bounding-box predictions to localize areas of new progression. Recall, precision, and mean average precision were used to evaluate the new progression localization. One-tailed paired t tests were used to assess statistical significance. Results The model outperformed most baselines in progression classification, achieving macro AUC scores of 0.72 ± 0.004 for atelectasis, 0.75 ± 0.007 for consolidation, 0.76 ± 0.017 for edema, 0.81 ± 0.006 for effusion, 0.7 ± 0.032 for pneumonia, and 0.69 ± 0.01 for pneumothorax. For new observation localization, the model achieved mean average precision scores of 0.25 ± 0.03 for atelectasis, 0.34 ± 0.03 for consolidation, 0.33 ± 0.03 for edema, and 0.31 ± 0.03 for pneumothorax. Conclusion Disease progression classification models were developed on a large chest radiograph dataset, which can be used to monitor interval changes and detect new pathologic conditions on chest radiographs. Keywords: Prognosis, Unsupervised Learning, Transfer Learning, Convolutional Neural Network (CNN), Emergency Radiology, Named Entity Recognition Supplemental material is available for this article. © RSNA, 2024 See also commentary by Alves and Venkadesh in this issue.

Keywords: Convolutional Neural Network (CNN); Emergency Radiology; Named Entity Recognition; Prognosis; Transfer Learning; Unsupervised Learning.

Figure 1:

An example of progression label extraction from radiology reports. A progression label is defined as a triplet of three entities: anatomy (“left apical”), observation (“pneumothorax”), and progression (“increase”).

Figure 2:

(A) Schematic for pretraining the Anatomy-Guided chest x-ray Network (AGXNet) (17). The network is pretrained using 69 weak labels, which include 46 labels denoting the absence or presence of abnormalities in anatomic landmarks and 23 labels denoting the absence or presence of observations on chest radiographs. A residual attention module captures the relationships between these labels. (B) Schematic for fine-tuning the twin AGXNet for progression classification. Anatomic saliency maps enable the encoder to attend to the same anatomic landmark across images. Symbols f_a and f_o denote the feature maps of the anatomy network and observation network, respectively. BCE = binary cross-entropy, CAM = class activation map, DNN = deep neural network.

Figure 3:

MIMIC-CXR data selection flowchart. The initial dataset had a total of 377 110 chest radiographs and their associated radiology reports from 65 379 patients. Only frontal view radiographs (243 008) were included for pretraining models, with 69 binary labels for each image. From this pretraining dataset, 176 168 images had prior images from 34 123 patients. A selection process was applied to choose images with progressions related to six observations and interval times shorter than 1 year, resulting in 28 153 frontal view radiographs with 36 743 progression labels (triplets) from 9737 patients for fine-tuning the models. The pretraining dataset was randomly split into training, validation, and testing sets in a ratio of 70:10:20 with no shared patients, and this split was kept consistent for model fine-tuning.

Figure 4:

Graphs compare models for progression classification on chest radiographs. Area under the receiver operating characteristic curve (AUC) was computed for each progression class using a one-vs-rest scheme, and the macro AUC represents the average AUC of the four classes. Bars display the mean and SD derived from five random data splits. Dots indicate the AUC values from each split. Figure values are included in Table S4. All models used DenseNet121 as the backbone. Random: initial twin network weights randomly initialized; ImageNet: twin network pretrained on ImageNet dataset; ConVIRT: twin network pretrained using ConVIRT (24) method; GLoRIA: twin network pretrained using GLoRIA (25) method; Anatomy-Guided chest x-ray Network (AGXNet): twin network pretrained using AGXNet (17).

Figure 5:

Graphs show ablation analysis for models with and without using anatomic saliency maps generated by the anatomy network. Bars display the mean and SD derived from five random data splits. Dots indicate the area under the receiver operating characteristic curve (AUC) values from each split. Figure values are included in Table S5. All models used DenseNet121 as the backbone. Random: initial twin network weights randomly initialized; ImageNet: twin network pretrained on ImageNet dataset; ConVIRT: twin network pretrained using ConVIRT (24) method; GLoRIA: twin network pretrained using GLoRIA (25) method; Anatomy-Guided chest x-ray Network (AGXNet): twin network pretrained using AGXNet (17).

Figure 6:

Visual evaluation of twin Anatomy-Guided chest x-ray Network (AGXNet) saliency maps. Column 1: sentences with progression labels (triplets); column 2: frontal chest radiographs; column 3: anatomy network class activation maps (CAMs); column 4: gradient-weighted class activation maps (GradCAM) for new pathologic conditions. Examples for (A) atelectasis, (B) consolidation, (C) edema, and (D) pneumothorax. Green boxes: reference standard anatomic landmarks; orange boxes: generated anatomic bounding box; red boxes: generated bounding box for new progression detection.