Automated Multi-View Multi-Modal Assessment of COVID-19 Patients Using Reciprocal Attention and Biomedical Transform

Abstract

Automated severity assessment of coronavirus disease 2019 (COVID-19) patients can help rationally allocate medical resources and improve patients' survival rates. The existing methods conduct severity assessment tasks mainly on a unitary modal and single view, which is appropriate to exclude potential interactive information. To tackle the problem, in this paper, we propose a multi-view multi-modal model to automatically assess the severity of COVID-19 patients based on deep learning. The proposed model receives multi-view ultrasound images and biomedical indices of patients and generates comprehensive features for assessment tasks. Also, we propose a reciprocal attention module to acquire the underlying interactions between multi-view ultrasound data. Moreover, we propose biomedical transform module to integrate biomedical data with ultrasound data to produce multi-modal features. The proposed model is trained and tested on compound datasets, and it yields 92.75% for accuracy and 80.95% for recall, which is the best performance compared to other state-of-the-art methods. Further ablation experiments and discussions conformably indicate the feasibility and advancement of the proposed model.

Keywords: COVID-19; computer aided diagnosis; deep learning; multi-modal; multi-view.

Figure 1

Typical examples of abnormal ultrasound cases related to coronavirus disease 2019 (COVID-19). (A) The pleural line is jagged or concave. (B) The pleural line is broken. (C) The scanning area shows a wide range of dense white areas, with or without large consolidation.

Figure 2

Overall flowchart of the proposed model. The proposed model receives multi-view ultrasound image pairs and biomedical indices of COVID-19 to conduct severity assessment tasks. The proposed reciprocal attention module tackles the multi-view ultrasound data and the proposed biomedical transform module tackles the biomedical data.

Figure 3

Detailed architecture of reciprocal attention module. Reciprocal attention module receives ultrasound image pairs and generates bidirectional attention features utilizing attention mechanism (33).

Figure 4

Detailed architectures of biomedical transform module. The biomedical transform module receives biomedical indices and generates parameters of affine transformation for ultrasonic features to obtain hybrid features.

Figure 5

Confuse matrices of severity assessment for COVID-19 patients. (A) VGG11BN (38), (B) DenseNet121 (40), (C) ResNet18 (39), (D) SENet (41), (E) SEResNet (41), (F) Xception (42), (G) InceptionV4 (43), (H) Sharifrazi et al. (26) and, (I) RAB (Ours).

Figure 6

Structures of baseline models. (A) Single view (SV) Model. (B) Dual view model without reciprocal attention module and biomedical transform module (DV) Model. (C) Reciprocal attention model without biomedical transform module (RA) Model. (D) Biomedical transform module before reciprocal attention module (named RAB-early)-early Model.

Figure 7

Confuse matrices of ablation experiments. (A) SV Model, (B) DV Model, (C) RA Model, (D) RAB-early Model, and (E) RAB-late Model.

Figure 8

Confuse Matrices of different loss functions. (A) BCE Loss model. (B) Focal Loss model.

Figure 9

Nidus-related visualization of ultrasound images. Using grad-CAM (44), the proposed model could highlight the image areas that are most relevant to COVID-19.

Figure 10

Statistical characteristics of biomedical indices. × stands for the mean value. The red and blue bar stands for the standard deviation for severe and mild cases, respectively. (A) Lymphocyte (absolute value). (B) C-reactive protein. (C) Lactate dehydrogenase. (D) Procalcitonin. (E) Interleukin-6.