Inter-rater reliability and prospective validation of a clinical prediction rule for SARS-CoV-2 infection

Abstract

Objectives: Accurate estimation of the risk of SARS-CoV-2 infection based on bedside data alone has importance to emergency department (ED) operations and throughput. The 13-item CORC (COVID [or coronavirus] Rule-out Criteria) rule had good overall diagnostic accuracy in retrospective derivation and validation. The objective of this study was to prospectively test the inter-rater reliability and diagnostic accuracy of the CORC score and rule (score ≤ 0 negative, > 0 positive) and compare the CORC rule performance with physician gestalt.

Methods: This noninterventional study was conducted at an urban academic ED from February 2021 to March 2021. Two practitioners were approached by research coordinators and asked to independently complete a form capturing the CORC criteria for their shared patient and their gestalt binary prediction of the SARS-CoV-2 test result and confidence (0%-100%). The criterion standard for SARS-CoV-2 was from reverse transcriptase polymerase chain reaction performed on a nasopharyngeal swab. The primary analysis was from weighted Cohen's kappa and likelihood ratios (LRs).

Results: For 928 patients, agreement between observers was good for the total CORC score, κ = 0.613 (95% confidence interval [CI] = 0.579-0.646), and for the CORC rule, κ = 0.644 (95% CI = 0.591-0.697). The agreement for clinician gestalt binary determination of SARs-CoV-2 status was κ = 0.534 (95% CI = 0.437-0.632) with median confidence of 76% (first-third quartile = 66-88.5). For 425 patients who had the criterion standard, a negative CORC rule (both observers scored CORC < 0), the sensitivity was 88%, and specificity was 51%, with a negative LR (LR-) of 0.24 (95% CI = 0.10-0.50). Among patients with a mean CORC score of >4, the prevalence of a positive SARS-CoV-2 test was 58% (95% CI = 28%-85%) and positive LR was 13.1 (95% CI = 4.5-37.2). Clinician gestalt demonstrated a sensitivity of 51% and specificity of 86% with a LR- of 0.57 (95% CI = 0.39-0.74).

Conclusion: In this prospective study, the CORC score and rule demonstrated good inter-rater reliability and reproducible diagnostic accuracy for estimating the pretest probability of SARs-CoV-2 infection.

Keywords: COVID-19; SARS-CoV-2; decision making; diagnosis; multivariable analysis; probability; prognosis; registries; risk.

FIGURE 1

Flow diagram of patient encounters

FIGURE 2

Bland‐Altman plot showing the difference in the score from the CORC score between two independent observers on the Y‐axis, plotted as a function of the average of the two reviewer scores on the X‐axis. The shaded rectangle contains 95% confidence limits for the Y‐axis data (−2.4 to +2.5). CORC, COVID‐19 Rule‐out Criteria

FIGURE 3

Receiver operating characteristic curve for the mean of the total CORC score from two observers for the criterion standard of a positive same‐day nucleic acid test on a nasopharyngeal swab for SARS‐CoV‐2. CORC, COVID‐19 Rule‐out Criteria