Effect of image resolution on automated classification of chest X-rays

Abstract

Purpose: Deep learning (DL) models have received much attention lately for their ability to achieve expert-level performance on the accurate automated analysis of chest X-rays (CXRs). Recently available public CXR datasets include high resolution images, but state-of-the-art models are trained on reduced size images due to limitations on graphics processing unit memory and training time. As computing hardware continues to advance, it has become feasible to train deep convolutional neural networks on high-resolution images without sacrificing detail by downscaling. This study examines the effect of increased resolution on CXR classification performance.

Approach: We used the publicly available MIMIC-CXR-JPG dataset, comprising 377,110 high resolution CXR images for this study. We applied image downscaling from native resolution to $2048\times 2048\text{pixels}$, $1024\times 1024\text{pixels}$, $512\times 512\text{pixels}$, and $256\times 256\text{pixels}$ and then we used the DenseNet121 and EfficientNet-B4 DL models to evaluate clinical task performance using these four downscaled image resolutions.

Results: We find that while some clinical findings are more reliably labeled using high resolutions, many other findings are actually labeled better using downscaled inputs. We qualitatively verify that tasks requiring a large receptive field are better suited to downscaled low resolution input images, by inspecting effective receptive fields and class activation maps of trained models. Finally, we show that stacking an ensemble across resolutions outperforms each individual learner at all input resolutions while providing interpretable scale weights, indicating that diverse information is extracted across resolutions.

Conclusions: This study suggests that instead of focusing solely on the finest image resolutions, multi-scale features should be emphasized for information extraction from high-resolution CXRs.

Keywords: chest X-ray; deep learning; image resolution; multitask classification; receptive field.

Fig. 1

Distribution of 2D image sizes in the MIMIC-CXR-JPG dataset. Marginal distributions are shown as histograms along the top and right axes. The scatter plot shows many actual native resolutions, but the marginal histograms show that these are concentrated in a few common resolutions: $\sim 2000\times 2000$, $2500\times 2500$, $2500\times 3000$, and $3000\times 2500\text{pixels}$.

Fig. 2

ERF of trained DenseNet121 models relative to image size, for different image resolutions. As the image resolution increases, ERF decreases due to corresponding decrease in pixel size.

Fig. 3

Workflow for the stacked ensemble model for a single input image. First, the image is resized to $256\times 256$, $512\times 512$, $1024\times 1024$, and $2048\times 2048$ resolution images and then passed to their corresponding fine-tuned models. The outputs from the models are then passed to a sigmoid function to get the output probabilities for each model. Finally, for each of the 14 labels, the predictions from the 4 models are multiplied by the learned weights to get the stacked ensemble prediction.

Fig. 4

(a) Image-level prediction where each image is treated separately and given a label corresponding to the radiology report of the study it belongs to. (b) Study-level prediction where images corresponding to a study are aggregated first and then given a label according to the radiology report of the study.

Fig. 5

Grad-CAM computed to visualize influential regions for predicting “cardiomegaly” (top), “pneumonia” (middle), and “pneumothorax” (bottom). Ground truth labels are shown overlaid on the original image, whereas for each Grad-CAM image, labels with probabilities $>50\%$ are shown. The top row shows that “cardiomegaly” is only correctly predicted at $256\times 256$ resolution, presumably due to ERFs at high resolution being too small to encompass the entire heart. The middle row is a perfect example of the effect of receptive field on resolution. As the resolution is increasing, the model is able to better predict “pneumonia.” But at resolution $2048\times 2048$, the ERF becomes too small to predict “pneumonia.” Conversely, the bottom row shows unreliable prediction of “pneumothorax” at coarse resolution due to lack of fine-scale information.

Fig. 6

Barplot showing average stacked DenseNet scale-ensemble weights for each resolution for each task. Tasks requiring mostly coarse-scale information, such as cardiomegaly, place larger weight on resolutions $256\times 256$ and $512\times 512$, whereas others such as support devices focus on higher resolutions.