The 2021 SIIM-FISABIO-RSNA Machine Learning COVID-19 Challenge: Annotation and Standard Exam Classification of COVID-19 Chest Radiographs

Abstract

We describe the curation, annotation methodology, and characteristics of the dataset used in an artificial intelligence challenge for detection and localization of COVID-19 on chest radiographs. The chest radiographs were annotated by an international group of radiologists into four mutually exclusive categories, including "typical," "indeterminate," and "atypical appearance" for COVID-19, or "negative for pneumonia," adapted from previously published guidelines, and bounding boxes were placed on airspace opacities. This dataset and respective annotations are available to researchers for academic and noncommercial use.

Keywords: Artificial Intelligence; COVID-19; Machine Learning; Pneumonia; Radiography; Thorax.

Fig. 1

The distribution of cases obtained from the Valencian Region Medical ImageBank (BIMCV) and RSNA International COVID-19 Open Radiology Database (RICORD). *The BIMCV data was obtained on 12/2020, and the current dataset is larger

Fig. 2

For near-confluent opacities, annotators were instructed to draw one larger more encompassing bounding box (b), rather than multiple smaller boxes (a) to improve standardization and decrease variability in the annotations. The radiographs above demonstrate bilateral airspace opacities in a COVID-19 positive patient

Fig. 3

Panel a shows an indeterminate appearance of COVID-19. There are bilateral central opacities, which are outlined by the blue bounding boxes. This pattern is compatible with cardiogenic pulmonary edema. The image also depicts the web-based annotation platform (MD.ai) used by the radiologists, and the exam-level annotation options. Panel b shows an indeterminate appearance of COVID-19. There are unilateral opacities outlined by the blue bounding box

Fig. 4

Typical appearances of COVID-19. Sample images (Panels a, b, c, d) of four CXRs demonstrating typical appearances of COVID-19, manifested by peripheral bilateral airspace opacities, which are outlined by blue bounding boxes. Panel e shows diffuse bilateral airspace opacities, both central and peripheral, as outlined by the blue bounding boxes. There is also mild reduction in lung volumes. These findings are commonly seen in severe COVID-19 in hospitalized patients with acute respiratory distress syndrome (ARDS)

Fig. 5

Atypical appearances of COVID-19. Panel a shows a left pleural effusion. Panel b shows a right upper lobe mass. Panel c shows a right pneumothorax (demarcated by the white arrows)