Artificial intelligence model on chest imaging to diagnose COVID-19 and other pneumonias: A systematic review and meta-analysis

Abstract

Objectives: When diagnosing Coronavirus disease 2019(COVID-19), radiologists cannot make an accurate judgments because the image characteristics of COVID-19 and other pneumonia are similar. As machine learning advances, artificial intelligence(AI) models show promise in diagnosing COVID-19 and other pneumonias. We performed a systematic review and meta-analysis to assess the diagnostic accuracy and methodological quality of the models.

Methods: We searched PubMed, Cochrane Library, Web of Science, and Embase, preprints from medRxiv and bioRxiv to locate studies published before December 2021, with no language restrictions. And a quality assessment (QUADAS-2), Radiomics Quality Score (RQS) tools and CLAIM checklist were used to assess the quality of each study. We used random-effects models to calculate pooled sensitivity and specificity, I² values to assess heterogeneity, and Deeks' test to assess publication bias.

Results: We screened 32 studies from the 2001 retrieved articles for inclusion in the meta-analysis. We included 6737 participants in the test or validation group. The meta-analysis revealed that AI models based on chest imaging distinguishes COVID-19 from other pneumonias: pooled area under the curve (AUC) 0.96 (95 % CI, 0.94-0.98), sensitivity 0.92 (95 % CI, 0.88-0.94), pooled specificity 0.91 (95 % CI, 0.87-0.93). The average RQS score of 13 studies using radiomics was 7.8, accounting for 22 % of the total score. The 19 studies using deep learning methods had an average CLAIM score of 20, slightly less than half (48.24 %) the ideal score of 42.00.

Conclusions: The AI model for chest imaging could well diagnose COVID-19 and other pneumonias. However, it has not been implemented as a clinical decision-making tool. Future researchers should pay more attention to the quality of research methodology and further improve the generalizability of the developed predictive models.

Keywords: 2D, two-dimensional; 3D, three-dimensional; AI, artificial intelligence; AUC, area under the curve; Artificial Intelligence; CNN, Convolutional neural network; COVID-19; COVID-19, Coronavirus disease 2019; CRP, C-reactive protein; CT, Computed tomography; CXR, Chest X-Ray; Diagnostic Imaging; GGO, ground-glass opacities; KNN, K-nearest neighbor; LASSO, least absolute shrinkage and selection operator; MEERS-COV, Middle East respiratory syndrome coronavirus; ML, machine learning; Machine learning; PLR, negative likelihood ratio; PLR, positive likelihood ratio; Pneumonia; ROI, regions of interest; RT-PCR, Reverse transcriptase polymerase chain reaction; SARS, severe acute respiratory syndrome; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; SROC, summary receiver operating characteristic; SVM, Support vector machine.

Fig. 1

Flow diagram of the study selection process for this meta-analysis.

Fig. 2

Methodological quality evaluated by using the Radiomics Quality Score (RQS) tool. (A). Proportion of studies with different RQS percentage score. (B). Average scores of each RQS item (gray bars stand for the full points of each item, and red bars show actual points).

Fig. 3

CLAIM items of the 19 included studies expressed as percentage of the ideal score according to the six key domains. CLAIM, Checklist for Artificial Intelligence in Medical Imaging.

Fig. 4

Coupled forest plots of pooled sensitivity and specificity of diagnostic performance of chest imaging for distinguished COVID-19 and other pneumonias. The numbers are pooled estimates with 95 % CIs in parentheses; horizontal lines indicate 95 % CIs.

Fig. 5

Diagnostic performance of SROC curve of an artificial intelligence model for distinguishing COVID-19 from other pneumonias on chest imaging. There was an obvious difference between the 95 % confidence and 95 % prediction regions, indicating a high possibility of heterogeneity across the studies.

Fig. 6

Effective sample size (ESS) funnel plots and the associated regression test of asymmetry, as reported by Deeks et al. A p value < 0.10 was considered evidence of asymmetry and potential publication bias.