Multi-modal deep learning methods for classification of chest diseases using different medical imaging and cough sounds

Abstract

Chest disease refers to a wide range of conditions affecting the lungs, such as COVID-19, lung cancer (LC), consolidation lung (COL), and many more. When diagnosing chest disorders medical professionals may be thrown off by the overlapping symptoms (such as fever, cough, sore throat, etc.). Additionally, researchers and medical professionals make use of chest X-rays (CXR), cough sounds, and computed tomography (CT) scans to diagnose chest disorders. The present study aims to classify the nine different conditions of chest disorders, including COVID-19, LC, COL, atelectasis (ATE), tuberculosis (TB), pneumothorax (PNEUTH), edema (EDE), pneumonia (PNEU). Thus, we suggested four novel convolutional neural network (CNN) models that train distinct image-level representations for nine different chest disease classifications by extracting features from images. Furthermore, the proposed CNN employed several new approaches such as a max-pooling layer, batch normalization layers (BANL), dropout, rank-based average pooling (RBAP), and multiple-way data generation (MWDG). The scalogram method is utilized to transform the sounds of coughing into a visual representation. Before beginning to train the model that has been developed, the SMOTE approach is used to calibrate the CXR and CT scans as well as the cough sound images (CSI) of nine different chest disorders. The CXR, CT scan, and CSI used for training and evaluating the proposed model come from 24 publicly available benchmark chest illness datasets. The classification performance of the proposed model is compared with that of seven baseline models, namely Vgg-19, ResNet-101, ResNet-50, DenseNet-121, EfficientNetB0, DenseNet-201, and Inception-V3, in addition to state-of-the-art (SOTA) classifiers. The effectiveness of the proposed model is further demonstrated by the results of the ablation experiments. The proposed model was successful in achieving an accuracy of 99.01%, making it superior to both the baseline models and the SOTA classifiers. As a result, the proposed approach is capable of offering significant support to radiologists and other medical professionals.

Fig 1. CSI, CXR, and CT scans are the three diagnostic tools that are indicated for use in the process of identifying a variety of chest disorders.

Fig 2. Sample CXR and CT scan images of multiple chest diseases.

Fig 3. Scalogram image of multiple chest diseases coughs sound.

Fig 4. Steps of conducting pre-processing.

Fig 5. Summary of the proposed models.

Fig 6. Generating CXR, CT scan, and CSI using MWDG.

(a) Rotation, (b) HST, (c) VST, (d) NI, (e) GCOR, (f) RTS, and (g) Scaling.

Fig 7. Training-Validation accuracy and loss.

(a) Vgg-19, (b) ResNet-101, (c) ResNet-50, (d) DenseNet-121, (e) Inception-V3, (f) EfficientNetB0, (g) DenseNet-201, and (h) Proposed P(4) model.

Fig 8. Confusion matrix.

(a) Vgg-19, (b) ResNet-101, (c) ResNet-50, (d) DenseNet-121, (e) Inception-V3, (f) Proposed P(4) model, (g) EfficientNetB0, and (h) DenseNet-201.

Fig 9. AU(ROC) for class wise evaluation of chest diseases.

(a) Vgg-19, (b) ResNet-101, (c) ResNet-50, (d) DenseNet-121, (e) Inception-V3, (f) Proposed P(4) model, (g) EfficientNetB0, and (h) DenseNet-201.

Fig 10. GRAD-CAM visualization of the proposed model for highlighting the infected region of nine chest diseases.