Performance characteristics of five immunoassays for SARS-CoV-2: a head-to-head benchmark comparison

Abstract

Background: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has caused a global pandemic in 2020. Testing is crucial for mitigating public health and economic effects. Serology is considered key to population-level surveillance and potentially individual-level risk assessment. However, immunoassay performance has not been compared on large, identical sample sets. We aimed to investigate the performance of four high-throughput commercial SARS-CoV-2 antibody immunoassays and a novel 384-well ELISA.

Methods: We did a head-to-head assessment of SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA), LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy), Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland), SARS-CoV-2 Total assay (Siemens, Munich, Germany), and a novel 384-well ELISA (the Oxford immunoassay). We derived sensitivity and specificity from 976 pre-pandemic blood samples (collected between Sept 4, 2014, and Oct 4, 2016) and 536 blood samples from patients with laboratory-confirmed SARS-CoV-2 infection, collected at least 20 days post symptom onset (collected between Feb 1, 2020, and May 31, 2020). Receiver operating characteristic (ROC) curves were used to assess assay thresholds.

Findings: At the manufacturers' thresholds, for the Abbott assay sensitivity was 92·7% (95% CI 90·2-94·8) and specificity was 99·9% (99·4-100%); for the DiaSorin assay sensitivity was 96·2% (94·2-97·7) and specificity was 98·9% (98·0-99·4); for the Oxford immunoassay sensitivity was 99·1% (97·8-99·7) and specificity was 99·0% (98·1-99·5); for the Roche assay sensitivity was 97·2% (95·4-98·4) and specificity was 99·8% (99·3-100); and for the Siemens assay sensitivity was 98·1% (96·6-99·1) and specificity was 99·9% (99·4-100%). All assays achieved a sensitivity of at least 98% with thresholds optimised to achieve a specificity of at least 98% on samples taken 30 days or more post symptom onset.

Interpretation: Four commercial, widely available assays and a scalable 384-well ELISA can be used for SARS-CoV-2 serological testing to achieve sensitivity and specificity of at least 98%. The Siemens assay and Oxford immunoassay achieved these metrics without further optimisation. This benchmark study in immunoassay assessment should enable refinements of testing strategies and the best use of serological testing resource to benefit individuals and population health.

Funding: Public Health England and UK National Institute for Health Research.

Figure 1

Sensitivity and specificity for each assay on samples taken ≥20 days post symptom onset in patients with laboratory-confirmed SARS-CoV-2 infection for positive samples, and >6 months before the first known COVID-19 cases for negative samples The UK MHRA performance target is shown (dark grey dashed line), including the required lower bound of the 95% CI (light grey dashed line) for sensitivity and specificity. Data are presented for 976 known-negative samples and 536 known-positive samples run on each assay. Equivocal results were excluded from the calculation of sensitivity and specificity for the DiaSorin assay (n=9). MHRA=UK Medicines and Healthcare products Regulatory Agency. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).

Figure 2

Distribution of numerical results obtained for each commercial assay Results are represented as (A) histograms, to enable assessment of the frequency of values, and (B) dotplots, to review scatter of values, especially around thresholds. Prespecified assay thresholds are shown as dashed lines. For the purposes of plotting values on a log scale, values of zero were set to the lowest non-zero value and results of greater or less than the largest or smallest values were truncated to the largest and smallest values. Data are presented for 976 known negative samples and 536 known positive samples run across all assays. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).

Figure 3

ROC curves for each assay at the specification of samples taken ≥20 post symptom onset The green shaded areas represent a target sensitivity and specificity of at least 98%. Dashed lines show the 98% sensitivity and specificity thresholds used as the standard. Assay threshold values associated with ten exemplar points on the ROC curve are shown in each panel. Data are presented for 976 known negative samples and 536 known positive samples run on each assay. ROC=receiver operating characteristic. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).

Figure 4

False negatives, false positives, and total errors per 1 million tests False negatives (A), false positives (B) and total errors (C) per 1 million tests, using the unadjusted thresholds (manufacturers, and Oxford immunoassay), and sensitivity and specificity for all assays for samples tested after at least 20 days post symptom onset, at population SARS-CoV-2 seroprevalence of 5%, 10%, 20%, and 50%. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).