The accuracy of teleradiologists in diagnosing COVID-19 based on a French multicentric emergency cohort

Abstract

Objectives: To evaluate the accuracy of diagnoses of COVID-19 based on chest CT as well as inter-observer agreement between teleradiologists during on-call duty and senior radiologists in suspected COVID-19 patients.

Materials and methods: From March 13, 2020, to April 14, 2020, consecutive suspected COVID-19 adult patients who underwent both an RT-PCR test and chest CT from 15 hospitals were included in this prospective study. Chest CTs were immediately interpreted by the on-call teleradiologist and were systematically blind reviewed by a senior radiologist. Readings were categorised using a five-point scale: (1) normal; (2) non-infectious findings; (3) infectious findings but not consistent with COVID-19 infection; (4) consistent with COVID-19 infection; and (5) typical appearance of COVID-19 infection. The diagnostic accuracy of chest CT and inter-observer agreement using the kappa coefficient were evaluated over the study period.

Results: In total, 513 patients were enrolled, of whom 244/513 (47.6%) tested positive for RT-PCR. First readings were scored 4 or 5 in 225/244 (92%) RT-PCR+ patients, and between 1 and 3 in 201/269 (74.7%) RT-PCR- patients. The data were highly consistent (weighted kappa = 0.87) and correlated with RT-PCR (p < 0.001, AUC_1st-reading = 0.89, AUC_2nd-reading = 0.93). The negative predictive value for scores of 4 or 5 was 0.91-0.92, and the PPV for a score of 5 was 0.89-0.96 at the first and second readings, respectively. Diagnostic accuracy was consistent over the study period, irrespective of a variable prevalence rate.

Conclusion: Chest CT demonstrated high diagnostic accuracy with strong inter-observer agreement between on-call teleradiologists with varying degrees of experience and senior radiologists over the study period.

Key points: • The accuracy of readings by on-call teleradiologists, relative to second readings by senior radiologists, demonstrated a sensitivity of 0.75-0.79, specificity of 0.92-0.97, NPV of 0.80-0.83, and PPV of 0.89-0.96, based on "typical appearance," as predictive of RT-PCR+. • Inter-observer agreement between the first reading in the emergency setting and the second reading by the senior emergency teleradiologist was excellent (weighted kappa = 0.87).

Keywords: COVID-19; Emergency service; Hospital; Polymerase chain reaction; Teleradiology; Tomography, X-ray computed.

Fig. 1

Flowchart

Fig. 2

Geographic distribution of patients included

Fig. 3

Categorisation using the 5-point scale adapted from the SFR. (a) Score of 1: normal chest CT. (b) Score of 2: lung abnormalities but non consistent with any pulmonary infection (showing acute cardiogenic pulmonary oedema with bilateral pleural effusion [black arrowheads]). (c) Score of 3: lung abnormalities consistent with an infection but not SARS-CoV-2 (showing a lobar consolidation [white arrowhead] with acute community-acquired pneumonia). (d) Score of 4: lung lesions compatible with COVID-19 (showing three small rounded central ground-glass opacities in the lower left lobe [white arrowheads]). (e) Score of 5: lung abnormalities strongly suspicious of COVID-19 (showing bilateral peripheral basal-predominant non-rounded ground-glass opacities [white arrowhead])

Fig. 4

Diagnostic accuracy measurements of on-call radiologists and inter-observer agreement variation between on-call radiologists and senior radiologists over the eight periods during the inclusion period: (a) number of cases of chest CT (depicted in black [weighted kappas for each step of the study period]) and number of RT-PCR+ patients (depicted in red [prevalence]). Accuracy corresponds to the percentage (from 0 to 1) of correctly classified observations, the p values for weighted kappa indices were all < 0.001; (b, c) diagnostic accuracy measures of on-call radiologists for a score of 4–5 (b) and 5 (c)