A hybrid explainable ensemble transformer encoder for pneumonia identification from chest X-ray images

Abstract

Introduction: Pneumonia is a microorganism infection that causes chronic inflammation of the human lung cells. Chest X-ray imaging is the most well-known screening approach used for detecting pneumonia in the early stages. While chest-Xray images are mostly blurry with low illumination, a strong feature extraction approach is required for promising identification performance.

Objectives: A new hybrid explainable deep learning framework is proposed for accurate pneumonia disease identification using chest X-ray images.

Methods: The proposed hybrid workflow is developed by fusing the capabilities of both ensemble convolutional networks and the Transformer Encoder mechanism. The ensemble learning backbone is used to extract strong features from the raw input X-ray images in two different scenarios: ensemble A (i.e., DenseNet201, VGG16, and GoogleNet) and ensemble B (i.e., DenseNet201, InceptionResNetV2, and Xception). Whereas, the Transformer Encoder is built based on the self-attention mechanism with multilayer perceptron (MLP) for accurate disease identification. The visual explainable saliency maps are derived to emphasize the crucial predicted regions on the input X-ray images. The end-to-end training process of the proposed deep learning models over all scenarios is performed for binary and multi-class classification scenarios.

Results: The proposed hybrid deep learning model recorded 99.21% classification performance in terms of overall accuracy and F1-score for the binary classification task, while it achieved 98.19% accuracy and 97.29% F1-score for multi-classification task. For the ensemble binary identification scenario, ensemble A recorded 97.22% accuracy and 97.14% F1-score, while ensemble B achieved 96.44% for both accuracy and F1-score. For the ensemble multiclass identification scenario, ensemble A recorded 97.2% accuracy and 95.8% F1-score, while ensemble B recorded 96.4% accuracy and 94.9% F1-score.

Conclusion: The proposed hybrid deep learning framework could provide promising and encouraging explainable identification performance comparing with the individual, ensemble models, or even the latest AI models in the literature. The code is available here: https://github.com/chiagoziemchima/Pneumonia_Identificaton.

Keywords: Chest X-ray imaging; Explainable artificial intelligence (XAI); Pneumonia identification; Self-attention network; Transfer ensemble learning; Transformer encoder (TE).

Graphical abstract

Fig. 1

The abstract organizational structure of the proposed hybrid deep learning framework for pneumonia identification from chest X-ray images. The pre-trained ensemble deep learning models serve as deep feature extractors, while the transformer encoder based on the self-attention mechanism and perceptron multilayer (MLP Block) is used for pneumonia accurate identification.

Fig. 2

Some samples of the deployed chest X-ray datasets. The first row depicts some images from the Mendeley dataset, while the second row depicts some images from the chest X-ray dataset.

Fig. 3

Diagrammatical illustration of the proposed Ensemble architecture.

Fig. 4

Visualization of a Multi-head self-attention Network and MLP blocks.

Fig. 5

Evaluation results of the proposed hybrid deep learning model in terms of ROC and PR curves for the binary classification scenario. Class 0 and 1 reflect the normal and pneumonia classes from Mendeley Dataset, respectively.

Fig. 6

Evaluation results of the proposed hybrid deep learning model in terms of ROC and PR curves for the multiclass identification scenario. Class 0, 1, and 2 reflect the bacteria pneumonia, normal, and viral pneumonia classes from Chest X-ray Dataset, respectively.

Fig. 7

The identification evaluation performance of the binary and multicalssification scenarios in terms of confusion matrices. (a), and (b) represent the binary classification confusion matrices of the Ensemble A model and the proposed hybrid deep learning model, respectively. Whereas, the (c) and (d) depict the multiclassifcation confusion matrices of the Ensemble A model and the proposed hybrid deep learning model, respectively.

Fig. 8

The visualization steps of the proposed transformer encoder model: (a) depicts the input chest X-ray image, (b) illustrates the divided input image into equal size non-overlapping patches, (c) shows learnable position embeddings of the input image patches, and (d) demonstrates the corresponding attention matrix.

Fig. 9

The transformer Encoder visualization based on the attention mechanism via the input chest X-ray image.

Fig. 10

Visual explainable heat maps (i.e., saliency maps) of the chest X-ray pneumonia image: (A) Depicts the heat maps of the pre-train DenseNet201and VGG16 models, (B) shows the heat map of the ensemble A deep learning model, and (C) illustrates the saliency map for the proposed hybrid deep learning framework.