Implementation of the Radiological Society of North America Expert Consensus Guidelines on Reporting Chest CT Findings Related to COVID-19: A Multireader Performance Study

Abstract

Background: RSNA expert consensus guidelines provide a framework for reporting CT findings related to COVID-19, but have had limited multireader validation.

Purpose: To assess the performance of the RSNA guidelines and quantify interobserver variability in application of the guidelines in patients undergoing chest CT for suspected COVID-19 pneumonia.

Materials and methods: A retrospective search from 1/15/20 to 3/30/20 identified 89 consecutive CT scans whose radiological report mentioned COVID-19. One positive or two negative RT-PCR tests for COVID-19 were considered the gold standard for diagnosis. Each chest CT scan was evaluated using RSNA guidelines by 9 readers (6 fellowship trained thoracic radiologists and 3 radiology resident trainees). Clinical information was obtained from the electronic medical record.

Results: There was strong concordance of findings between radiology training levels with agreement ranging from 60 to 86% among attendings and trainees (kappa 0.43 to 0.86). Sensitivity and specificity of "typical" CT findings for COVID-19 per the RSNA guidelines were on average 86% (range 72%-94%) and 80.2% (range 75-93%), respectively. Combined "typical" and "indeterminate" findings had a sensitivity of 97.5% (range 94-100%) and specificity of 54.7% (range 37-62%). A total of 163 disagreements were seen out of 801 observations (79.6% total agreement). Uncertainty in classification primarily derived from difficulty in ascertaining peripheral distribution, multiple dominant disease processes, or minimal disease.

Conclusion: The "typical appearance" category for COVID-19 CT reporting has an average sensitivity of 86% and specificity rate of 80%. There is reasonable interreader agreement and good reproducibility across various levels of experience.

Figure 1.

Examples of cases assigned the same RSNA consensus COVID-19 category by all readers. A) “Typical” category assigned to CT of 53-year-old man with COVID-19 pneumonia who presented after 2 weeks of cough, congestion, and fevers. Axial CT image shows multiple ground glass opacities with a peripheral predominance bilaterally, many with a round morphology. B) “Indeterminate” category assigned to CT of 82-year-old woman who presented with fever, exertional dyspnea, palpitations, and chest pain, with 2 PCRs negative for SARS-CoV-2. Axial CT image shows a small amount of ground glass opacity with a central predominance in the perihilar regions bilaterally. C) “Atypical” category assigned to CT of a 79-year-old woman who presented with fever, productive cough, dyspnea, and hypoxemia; 2 PCRs were negative for SARS-CoV-2. Axial CT image shows tree-in-bud nodules and consolidation in the lower lobes bilaterally, a pattern suggesting aspiration/pneumonia. D) “Negative for pneumonia” category assigned to CT of a 30-year-old woman who presented with one week of dry cough, sore throat, and severe fatigue; 2 PCRs were negative for SARS-CoV-2. Axial CT image shows a normal appearance of the lungs. Final diagnosis of symptoms was attributed to recurrent rheumatic myopericarditis within the context of her history of juvenile rheumatoid arthritis.

Figure 2.

Reasoning for Atypical/Indeterminate RSNA Score Among Attendings & Trainees. a) Distribution of scores among different readers. b) Percentage of cases with particular reasons for being assigned a category of indeterminate or atypical.

Figure 3.

Examples of cases for which there was significant disagreement in assignment of RSNA consensus COVID-19 category. A) 67-year-old man with clinical signs of pneumonia and 4 negative PCRs for COVID-19 with sputum samples positive for streptococcus pneumoniae. Axial CT image shows a combination of tree-in-bud centrilobular nodules in the lower lobes and peripheral ground glass opacity and consolidation in the left lower lobe. Categories 3, 2, and 1 were assigned by 4, 3, and 2 readers respectively. B) 23-year-old man with 2 negative PCR results for SARS-CoV-2 and presumed aspiration or non-23-COVID-19 infection. Axial CT image shows minimal patchy ground glass opacities in the left lower lobe; there was a question of atelectasis or subtle peripheral ground glass opacity in the posterior right lower lobe. Categories 3, 2, 1, and 0 were assigned by 1, 6, 1, and 1 readers respectively. C) 64-year-old woman with PCR-proven COVID-19 pneumonia who presented with fever, productive cough, fatigue, and anosmia. Axial CT image shows patchy ground glass opacities in the lingula and a small amount of peripheral ground glass opacity and atelectasis in the posterior lower lobes. Categories 3, 2, 1, and 0 were assigned by 4, 3, 1, and 1 readers respectively. Reasons given by readers for uncertainty included doubts about peripheral distribution, and difficulty in classification in the setting of minimal disease and posterior atelectasis. D) 65-year-old woman with PCR-proven COVID-19 pneumonia who presented with palpitations, back pain, and low-grade fevers. Axial CT image shows patchy ground glass opacities bilaterally. Categories 3 and 2 were assigned by 5 and 4 readers respectively. Reasons given by readers for uncertainty included difficulty in classifying as peripheral or diffuse and questionable morphology of the ground glass opacities.

Figure 4.

Uncertainty Among Attending & Trainees: a) Reasons for uncertainty among cases as a percentage of all cases reviewed. OP- organizing pneumonia b) Average certainty scores between attendings and trainees. c) Histogram of number of readers with scores discrepant from attending consensus. d) Average certainty score by RSNA categorization, * indicates statistical significance,- p<.05 (2-tailed t-test)