Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Mar;33(3):2096-2104.
doi: 10.1007/s00330-022-09172-w. Epub 2022 Oct 25.

Chest radiograph classification and severity of suspected COVID-19 by different radiologist groups and attending clinicians: multi-reader, multi-case study

Arjun Nair  1 , Alexander Procter  2 , Steve Halligan  3 , Thomas Parry  3 , Asia Ahmed  2 , Mark Duncan  2 , Magali Taylor  2 , Manil Chouhan  2 , Trevor Gaunt  2 , James Roberts  2 , Niels van Vucht  2 , Alan Campbell  2 , Laura May Davis  2 , Joseph Jacob  4 , Rachel Hubbard  2 , Shankar Kumar  2 , Ammaarah Said  2 , Xinhui Chan  5 , Tim Cutfield  5 , Akish Luintel  5 , Michael Marks  5 , Neil Stone  5 , Sue Mallet  3
Affiliations

Chest radiograph classification and severity of suspected COVID-19 by different radiologist groups and attending clinicians: multi-reader, multi-case study

Arjun Nair et al. Eur Radiol. 2023 Mar.

Abstract

Objectives: To quantify reader agreement for the British Society of Thoracic Imaging (BSTI) diagnostic and severity classification for COVID-19 on chest radiographs (CXR), in particular agreement for an indeterminate CXR that could instigate CT imaging, from single and paired images.

Methods: Twenty readers (four groups of five individuals)-consultant chest (CCR), general consultant (GCR), and specialist registrar (RSR) radiologists, and infectious diseases clinicians (IDR)-assigned BSTI categories and severity in addition to modified Covid-Radiographic Assessment of Lung Edema Score (Covid-RALES), to 305 CXRs (129 paired; 2 time points) from 176 guideline-defined COVID-19 patients. Percentage agreement with a consensus of two chest radiologists was calculated for (1) categorisation to those needing CT (indeterminate) versus those that did not (classic/probable, non-COVID-19); (2) severity; and (3) severity change on paired CXRs using the two scoring systems.

Results: Agreement with consensus for the indeterminate category was low across all groups (28-37%). Agreement for other BSTI categories was highest for classic/probable for the other three reader groups (66-76%) compared to GCR (49%). Agreement for normal was similar across all radiologists (54-61%) but lower for IDR (31%). Agreement for a severe CXR was lower for GCR (65%), compared to the other three reader groups (84-95%). For all groups, agreement for changes across paired CXRs was modest.

Conclusion: Agreement for the indeterminate BSTI COVID-19 CXR category is low, and generally moderate for the other BSTI categories and for severity change, suggesting that the test, rather than readers, is limited in utility for both deciding disposition and serial monitoring.

Key points: • Across different reader groups, agreement for COVID-19 diagnostic categorisation on CXR varies widely. • Agreement varies to a degree that may render CXR alone ineffective for triage, especially for indeterminate cases. • Agreement for serial CXR change is moderate, limiting utility in guiding management.

Keywords: Coronavirus; Diagnosis; Observer variation; X-ray.

PubMed Disclaimer

Conflict of interest statement

The authors of this manuscript declare relationships with the following companies:

AN reports a medical advisory role with Aidence BV, an artificial intelligence company; AN reports a consultation role with Faculty Science Limited, an artificial intelligence company.

SH and SM report grants from the National Institute for Health Research (NIHR) outside the submitted work.

JJ reports Consultancy fees from Boehringer Ingelheim, F. Hoffmann-La Roche, GlaxoSmithKline, NHSX; is on the Advisory Boards of Boehringer Ingelheim, F. Hoffmann-La Roche; has received lecture fees from Boehringer Ingelheim, F. Hoffmann-La Roche, Takeda; receives grant funding from GlaxoSmithKline; holds a UK patent (application number 2113765.8).

The other authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Figures

Fig. 1
Fig. 1
STARD flowchart showing the derivation of CXR reading dataset per reading group
Fig. 2
Fig. 2
Percentage agreement with consensus for individual BSTI categories for reader groups
Fig. 3
Fig. 3
Percentage agreement with consensus for BSTI severity classification for reader groups
Fig. 4
Fig. 4
Frequency charts showing agreement with consensus for score change using the BSTI severity classification (a) and the Covid-RALES for reader groups (b)
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Nair A, Rodrigues JCL, Hare S, Edey A, Devaraj A, Jacob J, et al. A British Society of Thoracic Imaging statement: considerations in designing local imaging diagnostic algorithms for the COVID-19 pandemic. Clin Radiol. 2020;75(5):329–334. doi: 10.1016/j.crad.2020.03.008. - DOI - PMC - PubMed
    1. Guan WJ, Zhong NS. Clinical characteristics of Covid-19 in China. Reply N Engl J Med. 2020;382(19):1861–1862. - PubMed
    1. Wong HYF, Lam HYS, Fong AH, Leung ST, Chin TW, Lo CSY, et al. Frequency and distribution of chest radiographic findings in patients positive for COVID-19. Radiology. 2020;296(2):E72–EE8. doi: 10.1148/radiol.2020201160. - DOI - PMC - PubMed
    1. Liang W, Liang H, Ou L, Chen B, Chen A, Li C, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. 2020;180(8):1081–1089. doi: 10.1001/jamainternmed.2020.2033. - DOI - PMC - PubMed
    1. Public Health England(2020) COVID-19: investigation and initial clinical management of possible cases [updated 12/14/2020. Available from: https://www.gov.uk/government/publications/wuhan-novel-coronavirus-initi.... Accessed 23 Dec 2021