Laboratory evaluation of 185 commercial assays for detecting SARS-CoV-2: the UK response for mass testing

Abstract

Background: In August 2020, Public Health England and Oxford University were commissioned to design and deliver (with NHS Test and Trace, NHSTT) a rapid evaluation programme of antigen Lateral Flow Devices (LFDs) for SARS-CoV-2 for mass community testing.

Methods: A three-phase evaluation process was established: 1) desktop review of kits including claimed performance and supply; 2) laboratory testing with laboratory-grown SARS-CoV-2 virus and SARS-CoV-2 virus PCR negative volunteer samples; and 3) larger-scale laboratory testing of SARS-CoV-2 PCR positive and negative clinical samples. Variant of Concern (VOC) identification in the UK (December 2020), expanded laboratory methodology. Processes also evolved to improve workflow (irradiated viral stocks, dilution matrices, sample volumes, and replicates).

Findings: Overall, 1017 kits were screened at phase 1, 185 kits tested at phase 2 and 91 at phase 3. Sixteen kits failed phase 3 due to poor performance and eight more failed to detect VOC satisfactorily. Sixty-four kits were redesigns of previously failed kits. The overall pass rate for the laboratory evaluation was 35% and 5 kits were procured for the UK National Covid 19 Testing Programme.

Interpretation: The evaluation results had potential, time limited commercially sensitive aspects, and public sharing was limited to kits passing phase 3. Until now, the full data set has not been published. Over 2.5 billion self-test kits were deployed by the UK government following purchasing decisions informed by this work. We offer a potential blueprint for future evaluation programmes that might be required to assess LFDs to detect cases of a pandemic novel pathogen.

Funding: This study was funded by UK Department of Health and Social Care (UKHSA); UK Health Security Agency (formerly Public Health England and the National Health Service Test and Trace); and the University of OxfordNIHRBiomedical Research Centre.

Keywords: Coronavirus; LFD; Laboratory evaluation; Lateral flow device; Mass-testing; Pandemic; Public health; SARS-CoV-2.

Fig. 1

Phase 2 testing results. a) Summary of overall phase 2 testing results showing number and percentage of kits passing or failing, including reason for failure (n = 185); b) Timeline of phase 2 testing results showing changes to methodology.

Fig. 2

Comparison of sensitivities from phase 3 testing (clinical swab samples in VTM) over the duration of the evaluation as predicted proportion of infectiousness in 18291 test set. Numbers represent different LFD listed in Supplementary Table S1, red results indicate resubmitted LFD kits with Tween buffer.

Fig. 3

Relationship of viral concentration with intensity of LFD bands for the five kits purchased by the UK government. Individual graph title number = kit reference listed in Supplementary Table S1, x-axis 0–7 = LFD band intensity values (negative to very strong) N = number of LFDs with that band value in phase 3 sensitivity testing, y-axis = viral concentration (log RNA copies/ml), dashed line represents phase 2 sensitivity pass cut-off. Higher band values with lower viral concentration indicate more sensitive tests. Box plot shows the median, IQR, the whiskers are 1.5∗IQR and the outliers are outside the 1.5∗IQR range.

Fig. 4

Summary of number of LFD kits tested at each evaluation stage.