COVID-19 Hierarchical Classification Using a Deep Learning Multi-Modal

Abstract

Coronavirus disease 2019 (COVID-19), originating in China, has rapidly spread worldwide. Physicians must examine infected patients and make timely decisions to isolate them. However, completing these processes is difficult due to limited time and availability of expert radiologists, as well as limitations of the reverse-transcription polymerase chain reaction (RT-PCR) method. Deep learning, a sophisticated machine learning technique, leverages radiological imaging modalities for disease diagnosis and image classification tasks. Previous research on COVID-19 classification has encountered several limitations, including binary classification methods, single-feature modalities, small public datasets, and reliance on CT diagnostic processes. Additionally, studies have often utilized a flat structure, disregarding the hierarchical structure of pneumonia classification. This study aims to overcome these limitations by identifying pneumonia caused by COVID-19, distinguishing it from other types of pneumonia and healthy lungs using chest X-ray (CXR) images and related tabular medical data, and demonstrate the value of incorporating tabular medical data in achieving more accurate diagnoses. Resnet-based and VGG-based pre-trained convolutional neural network (CNN) models were employed to extract features, which were then combined using early fusion for the classification of eight distinct classes. We leveraged the hierarchal structure of pneumonia classification within our approach to achieve improved classification outcomes. Since an imbalanced dataset is common in this field, a variety of versions of generative adversarial networks (GANs) were used to generate synthetic data. The proposed approach tested in our private datasets of 4523 patients achieved a macro-avg F1-score of 95.9% and an F1-score of 87.5% for COVID-19 identification using a Resnet-based structure. In conclusion, in this study, we were able to create an accurate deep learning multi-modal to diagnose COVID-19 and differentiate it from other kinds of pneumonia and normal lungs, which will enhance the radiological diagnostic process.

Keywords: COVID-19; CXR; artificial intelligence; deep learning; diagnosis; hierarchical; image classification; multi-classes; multi-modal; pneumonia.

Figure 1

Proposed hierarchal class structure of pneumonia.

Figure 2

Proposed framework for multi-modal classification of CXR images and tabular data.

Figure 3

CXR samples of the dataset.

Figure 4

Exploratory analysis of the numeric features.

Figure 5

Correlation of the continuous features in the dataset.

Figure 6

An example of the rib shadow elimination process and result.

Figure 7

Samples of synthetic CXR images.

Figure 8

VGG-like multi-modal architecture.

Figure 9

VGG-backbone multi-modal architecture.

Figure 10

ResNet-like multi-modal architecture.

Figure 11

ResNet-backbone multi-modal architecture.

Figure 12

Comparison of macro-average accuracy for all models with and without a second dataset.

Figure 13

Testing accuracy (one-fold) against the number of epochs for the Resnet-backbone multi-modal in the last experiments.

Figure 14

Training and testing loss against the number of epochs for each multi-modal in the last experiments.

Figure 15

Confusion matrix for each multi-modal in the last experiments.

Figure 16

Macro-average ROC curve across all decisions for each multi-modal in the last experiments.