Comparison of Chest CT Grading Systems in Coronavirus Disease 2019 (COVID-19) Pneumonia

Abstract

Purpose: To compare the performance and interobserver agreement of the COVID-19 Reporting and Data System (CO-RADS), the COVID-19 imaging reporting and data system (COVID-RADS), the RSNA expert consensus statement, and the British Society of Thoracic Imaging (BSTI) guidance statement.

Materials and methods: In this case-control study, total of 100 symptomatic patients suspected of having COVID-19 were included: 50 patients with COVID-19 (59±17 years, 38 men) and 50 patients without COVID-19 (65±24 years, 30 men). Eight radiologists independently scored chest CT images of the cohort according to each reporting system. The area under the receiver operating characteristic curves (AUC) and interobserver agreements were calculated and statistically compared across the systems.

Results: A total of 800 observations were made for each system. The level of suspicion of COVID-19 correlated with the RT-PCR positive rate except for the "negative for pneumonia" classifications in all the systems (Spearman's coefficient: ρ=1.0, P=<.001 for all the systems). Average AUCs were as follows: CO-RADS, 0.84 (95% confidence interval, 0.83-0.85): COVID-RADS, 0.80 (0.78-0.81): the RSNA statement, 0.81 (0.79-0.82): and the BSTI statement, 0.84 (0.812-0.86). Average Cohen's kappa across observers was 0.62 (95% confidence interval, 0.58-0.66), 0.63 (0.58-0.68), 0.63 (0.57-0.69), and 0.61 (0.58-0.64) for CO-RADS, COVID-RADS, the RSNA statement and the BSTI statement, respectively. CO-RADS and the BSTI statement outperformed COVID-RADS and the RSNA statement in diagnostic performance (P=.<.05 for all the comparison).

Conclusions: CO-RADS, COVID-RADS, the RSNA statement and the BSTI statement provided reasonable performances and interobserver agreements in reporting CT findings of COVID-19.

Figure 1.

Flow chart illustrating the patient selection.

Figure 2a.

Cases illustrative of downgrading in COVID-RADS despite typical patterns seen in COVID-19. (a) An axial chest CT image of a 72-year-old male with COVID-19 shows peripheral, bilateral multifocal GGOs (arrows) with a background of mild emphysema. The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (b) An axial chest CT image of a 76-year-old female with COVID-19 shows peripheral, bilateral, multifocal GGOs with left pleural effusion (arrow). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (c) An axial chest CT image of a 49-year-old male with COVID-19 shows peripheral, bilateral, multifocal GGOs with mild mediastinal lymph node enlargements (arrows). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement.

Figure 2b.

Cases illustrative of downgrading in COVID-RADS despite typical patterns seen in COVID-19. (a) An axial chest CT image of a 72-year-old male with COVID-19 shows peripheral, bilateral multifocal GGOs (arrows) with a background of mild emphysema. The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (b) An axial chest CT image of a 76-year-old female with COVID-19 shows peripheral, bilateral, multifocal GGOs with left pleural effusion (arrow). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (c) An axial chest CT image of a 49-year-old male with COVID-19 shows peripheral, bilateral, multifocal GGOs with mild mediastinal lymph node enlargements (arrows). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement.

Figure 2c.

Cases illustrative of downgrading in COVID-RADS despite typical patterns seen in COVID-19. (a) An axial chest CT image of a 72-year-old male with COVID-19 shows peripheral, bilateral multifocal GGOs (arrows) with a background of mild emphysema. The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (b) An axial chest CT image of a 76-year-old female with COVID-19 shows peripheral, bilateral, multifocal GGOs with left pleural effusion (arrow). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (c) An axial chest CT image of a 49-year-old male with COVID-19 shows peripheral, bilateral, multifocal GGOs with mild mediastinal lymph node enlargements (arrows). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement.

Figure 3a.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Figure 3b.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Figure 3c.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Figure 3d.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Figure 4a.

Cases illustrative of disaccord in gradings between each set of criteria. (a) An axial chest CT image of a 24-year-old male without COVID-19 shows segmental GGOs and consolidation with bronchial wall thickening (dotted circle). The consensus grading was CO-RADS 2, COVID-RADS 3, indeterminate appearance in the RSNA expert consensus statement, indeterminate in the BSTI guidance statement. As illustrated in this case, if observers consider the findings compatible with typical bronchopneumonia, CO-RADS enables classification as a lower grade than do the other three sets of criteria. (b) An axial chest CT image of a 35-year-old female with COVID-19 shows multifocal peribronchovascular segmental centrilobular GGOs with bronchial wall thickening (dotted circle). There were centrilobular GGOs on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2B, atypical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. As illustrated in this case, lesions with centrilobular distribution may be underestimated in CO-RADS and the RSNA expert consensus statement. (c) An axial chest CT image of a 28-year-old male with COVID-19 shows peripheral, multifocal rounded GGOs with visible intralobular lines in the left lower lobe (dotted circle). The consensus grading was CO-RADS 4, COVID-RADS 3, typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. As illustrated in this case, CO-RADS classifies lesions with unilateral distribution as grade 4 despite their typical appearance seen in COVID-19, while the other three sets of criteria assigning it to the respective highest grade.

Figure 4b.

Cases illustrative of disaccord in gradings between each set of criteria. (a) An axial chest CT image of a 24-year-old male without COVID-19 shows segmental GGOs and consolidation with bronchial wall thickening (dotted circle). The consensus grading was CO-RADS 2, COVID-RADS 3, indeterminate appearance in the RSNA expert consensus statement, indeterminate in the BSTI guidance statement. As illustrated in this case, if observers consider the findings compatible with typical bronchopneumonia, CO-RADS enables classification as a lower grade than do the other three sets of criteria. (b) An axial chest CT image of a 35-year-old female with COVID-19 shows multifocal peribronchovascular segmental centrilobular GGOs with bronchial wall thickening (dotted circle). There were centrilobular GGOs on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2B, atypical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. As illustrated in this case, lesions with centrilobular distribution may be underestimated in CO-RADS and the RSNA expert consensus statement. (c) An axial chest CT image of a 28-year-old male with COVID-19 shows peripheral, multifocal rounded GGOs with visible intralobular lines in the left lower lobe (dotted circle). The consensus grading was CO-RADS 4, COVID-RADS 3, typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. As illustrated in this case, CO-RADS classifies lesions with unilateral distribution as grade 4 despite their typical appearance seen in COVID-19, while the other three sets of criteria assigning it to the respective highest grade.

Figure 4c.

Cases illustrative of disaccord in gradings between each set of criteria. (a) An axial chest CT image of a 24-year-old male without COVID-19 shows segmental GGOs and consolidation with bronchial wall thickening (dotted circle). The consensus grading was CO-RADS 2, COVID-RADS 3, indeterminate appearance in the RSNA expert consensus statement, indeterminate in the BSTI guidance statement. As illustrated in this case, if observers consider the findings compatible with typical bronchopneumonia, CO-RADS enables classification as a lower grade than do the other three sets of criteria. (b) An axial chest CT image of a 35-year-old female with COVID-19 shows multifocal peribronchovascular segmental centrilobular GGOs with bronchial wall thickening (dotted circle). There were centrilobular GGOs on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2B, atypical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. As illustrated in this case, lesions with centrilobular distribution may be underestimated in CO-RADS and the RSNA expert consensus statement. (c) An axial chest CT image of a 28-year-old male with COVID-19 shows peripheral, multifocal rounded GGOs with visible intralobular lines in the left lower lobe (dotted circle). The consensus grading was CO-RADS 4, COVID-RADS 3, typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. As illustrated in this case, CO-RADS classifies lesions with unilateral distribution as grade 4 despite their typical appearance seen in COVID-19, while the other three sets of criteria assigning it to the respective highest grade.

Figure 5a.

Cases illustrative of diagnostic challenges with severe pre-existing pulmonary abnormalities. (a) An axial chest CT image of a 91-year-old male without COVID-19 shows peripheral multifocal GGOs (dotted circles) in the background of usual interstitial pneumonia and moderate emphysema. There was a definite honeycomb lung destruction on other images (not shown). The consensus grading was CO-RADS 4, COVID-RADS 2B, typical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. (b) An axial chest CT image of a 57-year-old male with COVID-19 shows severe emphysema and airspace expansion with fibrosis. The consensus grading was CO-RADS 1, COVID-RADS 1, negative appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement.

Figure 5b.

Cases illustrative of diagnostic challenges with severe pre-existing pulmonary abnormalities. (a) An axial chest CT image of a 91-year-old male without COVID-19 shows peripheral multifocal GGOs (dotted circles) in the background of usual interstitial pneumonia and moderate emphysema. There was a definite honeycomb lung destruction on other images (not shown). The consensus grading was CO-RADS 4, COVID-RADS 2B, typical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. (b) An axial chest CT image of a 57-year-old male with COVID-19 shows severe emphysema and airspace expansion with fibrosis. The consensus grading was CO-RADS 1, COVID-RADS 1, negative appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement.