DR-MIL: deep represented multiple instance learning distinguishes COVID-19 from community-acquired pneumonia in CT images

Abstract

Background and objective: Given that the novel coronavirus disease 2019 (COVID-19) has become a pandemic, a method to accurately distinguish COVID-19 from community-acquired pneumonia (CAP) is urgently needed. However, the spatial uncertainty and morphological diversity of COVID-19 lesions in the lungs, and subtle differences with respect to CAP, make differential diagnosis non-trivial.

Methods: We propose a deep represented multiple instance learning (DR-MIL) method to fulfill this task. A 3D volumetric CT scan of one patient is treated as one bag and ten CT slices are selected as the initial instances. For each instance, deep features are extracted from the pre-trained ResNet-50 with fine-tuning and represented as one deep represented instance score (DRIS). Each bag with a DRIS for each initial instance is then input into a citation k-nearest neighbor search to generate the final prediction. A total of 141 COVID-19 and 100 CAP CT scans were used. The performance of DR-MIL is compared with other potential strategies and state-of-the-art models.

Results: DR-MIL displayed an accuracy of 95% and an area under curve of 0.943, which were superior to those observed for comparable methods. COVID-19 and CAP exhibited significant differences in both the DRIS and the spatial pattern of lesions (p<0.001). As a means of content-based image retrieval, DR-MIL can identify images used as key instances, references, and citers for visual interpretation.

Conclusions: DR-MIL can effectively represent the deep characteristics of COVID-19 lesions in CT images and accurately distinguish COVID-19 from CAP in a weakly supervised manner. The resulting DRIS is a useful supplement to visual interpretation of the spatial pattern of lesions when screening for COVID-19.

Keywords: COVID-19; Community-acquired pneumonia; Convolutional neural network; Deep learning; Lung CT image; Multiple instance learning.

Fig. 1

Overview of procedures used in this study.

Fig. 2

Pre-trained backbone networks (ResNet-50 and Xception) with fine-tuning for feature extraction.

Fig. 3

Four categories of comparable methods (2D voting, 2D montage, 3D down-sampling, and MIL with max or Noisy-AND pooling).

Fig. 4

Pseudo code representations for three comparable methods.

Fig. 5

Training and testing procedures of the proposed models by 10-fold cross-validation.

Fig. 6

ROC curves and AUC values for the proposed and comparable models.

Fig. 7

Confusion matrices for the proposed and comparable models: (a) M_{ResNet-50-MIL}, (b) M_{ResNet-50-MIL-frozen-layers}, (c) M_{ResNet-50-Voting}, (d) M_{ResNet-50-Montages}, (e) M_MedicalNet, (f) M_Xception-MIL, (g) M_{Xception-Voting}, (h) M_{Xception-Montages}, (i) M_{ResNet-50-MIL-max-pooling}, and (j) M_{ResNet-50-MIL-Noisy-AND-pooling}.

Fig. 8

Feature maps for ResNet-50 before and after fine-tuning.

Fig. 9

Distribution of deep features and DRIS for COVID-19 and CAP: (a) deep features for COVID-19 versus CAP, (b) DRIS for COVID-19 versus CAP, (c) DRIS for COVID-19 versus bacterial CAP, (d) DRIS for COVID-19 versus viral CAP, (e) DRIS for COVID-19 versus mycoplasma CAP, and (f) DRIS for COVID-19 versus CAP of uncertain etiology.

Fig. 10

Grad-CAM results and spatial patterns of COVID-19 and CAP lesions: (a) example of Grad-CAM for one case of COVID-19 and (b) comparison of spatial patterns of lesions between COVID-19 and CAP.

Fig. 11

Visual interpretation of used instances and the corresponding instances in the bags regarded as references and citers.