Facilitating standardized COVID-19 suspicion prediction based on computed tomography radiomics in a multi-demographic setting

Abstract

Objective: To develop an automatic COVID-19 Reporting and Data System (CO-RADS)-based classification in a multi-demographic setting.

Methods: This multi-institutional review boards-approved retrospective study included 2720 chest CT scans (mean age, 58 years [range 18-100 years]) from Italian and Russian patients. Three board-certified radiologists from three countries assessed randomly selected subcohorts from each population and provided CO-RADS-based annotations. CT radiomic features were extracted from the selected subcohorts after preprocessing steps like lung lobe segmentation and automatic noise reduction. We compared three machine learning models, logistic regression (LR), multilayer perceptron (MLP), and random forest (RF) for the automated CO-RADS classification. Model evaluation was carried out in two scenarios, first, training on a mixed multi-demographic subcohort and testing on an independent hold-out dataset. In the second scenario, training was done on a single demography and externally validated on the other demography.

Results: The overall inter-observer agreement for the CO-RADS scoring between the radiologists was substantial (k = 0.80). Irrespective of the type of validation test scenario, suspected COVID-19 CT scans were identified with an accuracy of 84%. SHapley Additive exPlanations (SHAP) interpretation showed that the "wavelet_(LH)_GLCM_Imc1" feature had a positive impact on COVID prediction both with and without noise reduction. The application of noise reduction improved the overall performance between the classifiers for all types.

Conclusion: Using an automated model based on the COVID-19 Reporting and Data System (CO-RADS), we achieved clinically acceptable performance in a multi-demographic setting. This approach can serve as a standardized tool for automated COVID-19 assessment.

Keypoints: • Automatic CO-RADS scoring of large-scale multi-demographic chest CTs with mean AUC of 0.93 ± 0.04. • Validation procedure resembles TRIPOD 2b and 3 categories, enhancing the quality of experimental design to test the cross-dataset domain shift between institutions aiding clinical integration. • Identification of COVID-19 pneumonia in the presence of community-acquired pneumonia and other comorbidities with an AUC of 0.92.

Keywords: COVID-19; Deep learning; Diagnostic imaging; SARS-CoV-2; Tomography X-ray computed.

Fig. 1

The overall workflow followed to validate machine learning models for automated COVID-19 suspicion staging based on the CO-RADS protocol. The scans with COVID-19 suspicion were selected from the Italian and Russian subcohorts retrospectively and annotated by experienced radiologists from different countries. In the first step, the datasets were processed using deep learning–based noise reduction (DLNR) and 3D segmentation masks were generated for each scan. Next, radiomic features were extracted and classified using ML algorithms. In the final step, statistical evaluation of the standard performance metrics and visualization of class-specific features were carried out to enhance the explainability of the models

Fig. 2

Data flowchart of Russian and Italian subcohorts included for the study with the training and test split. Below the flowchart, a description of different evaluation settings of data scenarios is depicted. Note that n refers to the number of patients. CO-RADS–COVID-19 Reporting and Data Systems, CT 0, 1, 2, 3, 4–severity–based Russian annotations

Fig. 3

The receiver operating curves of machine learning algorithms showing their ability to classify normal chest CT from other CO-RADS stages on the hold-out dataset (scenario 1). The performance of the classifiers increased after noise reduction

Fig. 4

Top twenty feature visualization using SHAP for CO-RADS classification in each data setting. The features are arranged in a descending order of feature importance (SHAP values). Using this feature importance map, one can observe how each feature contributes to the machine learning model’s predictions and identifies the common features

Fig. 5

The class-specific feature summery plot for CO-RADS 5 using SHAP in each setting. The feature impact on the classification is observed by a positive SHAP value indicated by red color. For example, “wavelet (LH) GLCM Imc1” shows positive impact on CO-RADS 5 prediction in most of the settings