Test performance evaluation of SARS-CoV-2 serological assays

doi:10.1101/2020.04.25.20074856

This is a preprint.

It has not yet been peer reviewed by a journal.

The National Library of Medicine is running a pilot to include preprints that result from research funded by NIH in PMC and PubMed.

[Preprint]. 2020 May 17:2020.04.25.20074856.

doi: 10.1101/2020.04.25.20074856.

Test performance evaluation of SARS-CoV-2 serological assays

Jeffrey D Whitman¹, Joseph Hiatt^{2

3

4

5

6

7}, Cody T Mowery^{2

3

5

6

7}, Brian R Shy¹, Ruby Yu^{5

7}, Tori N Yamamoto^{5

6

7}, Ujjwal Rathore^{4

5

6

7}, Gregory M Goldgof¹, Caroline Whitty^{1

5

7}, Jonathan M Woo^{5

6

7}, Antonia E Gallman^{2

5

8}, Tyler E Miller⁹, Andrew G Levine¹, David N Nguyen^{5

6

10}, Sagar P Bapat^{1

5

7}, Joanna Balcerek¹, Sophia A Bylsma¹¹, Ana M Lyons¹², Stacy Li¹², Allison Wai-Yi Wong², Eva Mae Gillis-Buck¹³, Zachary B Steinhart^{5

7}, Youjin Lee⁵, Ryan Apathy^{5

6

7}, Mitchell J Lipke^{5

7}, Jennifer Anne Smith^{7

14}, Tina Zheng^{2

3

15

16}, Ian C Boothby^{2

17}, Erin Isaza^{2

18}, Jackie Chan⁵, Dante D Acenas 2nd⁵, Jinwoo Lee^{2

19}, Trisha A Macrae^{2

19}, Than S Kyaw^{2

5}, David Wu^{2

3}, Dianna L Ng^{16

20}, Wei Gu¹, Vanessa A York²¹, Haig Alexander Eskandarian²¹, Perri C Callaway^{21

22}, Lakshmi Warrier²¹, Mary E Moreno²¹, Justine Levan²¹, Leonel Torres²¹, Lila A Farrington²¹, Rita Loudermilk²³, Kanishka Koshal²³, Kelsey C Zorn²⁴, Wilfredo F Garcia-Beltran⁹, Diane Yang⁹, Michael G Astudillo⁹, Bradley E Bernstein⁹, Jeffrey A Gelfand²⁵, Edward T Ryan²⁵, Richelle C Charles²⁵, A John Iafrate⁹, Jochen K Lennerz⁹, Steve Miller¹, Charles Y Chiu^{1

10

26}, Susan L Stramer²⁷, Michael R Wilson^{3

23}, Aashish Manglik²⁸, Chun Jimmie Ye^{29

30

31

32

33

34}, Nevan J Krogan^{4

35

36}, Mark S Anderson⁷, Jason G Cyster^{5

8}, Joel D Ernst²¹, Alan H B Wu¹, Kara L Lynch¹, Caryn Bern³⁴, Patrick D Hsu^{6

11}, Alexander Marson^{4

5

6

7

14

16

29

30

31}

Affiliations

¹ Department of Laboratory Medicine, University of California, San Francisco, San Francisco, CA 94143, USA.
² Medical Scientist Training Program, University of California, San Francisco, CA 94143, USA.
³ Biomedical Sciences Graduate Program, University of California, San Francisco, CA 94143, USA.
⁴ J. David Gladstone Institutes, San Francisco, CA 94158, USA.
⁵ Department of Microbiology and Immunology, University of California, San Francisco, CA 94143, USA.
⁶ Innovative Genomics Institute, University of California, Berkeley, CA 94720, USA.
⁷ Diabetes Center, University of California, San Francisco, San Francisco, CA 94143, USA.
⁸ Howard Hughes Medical Institute, University of California, San Francisco.
⁹ Department of Pathology, Massachusetts General Hospital/Harvard Medical School, Boston, MA, USA.
¹⁰ Department of Medicine, Division of Infectious Diseases, University of California, San Francisco, San Francisco, CA 94143, USA.
¹¹ Department of Bioengineering, University of California, Berkeley, Berkeley CA 94720 USA.
¹² Department of Integrative Biology, University of California, Berkeley, Berkeley CA 94720 USA.
¹³ Department of Surgery, University of California, San Francisco, CA 94143, USA.
¹⁴ Department of Medicine, University of California, San Francisco, San Francisco, CA 94143, USA.
¹⁵ Department of Neurology, University of California, San Francisco, CA 94158, USA.
¹⁶ Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco, San Francisco, CA 94158, USA.
¹⁷ Department of Dermatology, University of California, San Francisco, San Francisco, CA 94158, USA.
¹⁸ Program in Quantitative Biology, University of California, San Francisco, San Francisco, CA 94158, USA.
¹⁹ School of Medicine, University of California, San Francisco, San Francisco, CA 94158, USA.
²⁰ Department of Pathology, University of California, San Francisco, San Francisco, CA, 94158, USA.
²¹ Division of Experimental Medicine, Department of Medicine, University of California San Francisco, San Francisco CA, USA.
²² Infectious Diseases and Immunity Graduate Group, University of California Berkeley, Berkeley, CA, USA.
²³ Weill Institute for Neurosciences, Department of Neurology, University of California, San Francisco, San Francisco, CA.
²⁴ Department of Biochemistry and Biophysics, University of California, San Francisco, San Francisco, CA 94158, USA.
²⁵ Division of Infectious Diseases, Massachusetts General Hospital/Harvard Medical School, Boston, MA, USA.
²⁶ UCSF-Abbott Viral Diagnostics and Discovery Center, San Francisco, CA, USA.
²⁷ Scientific Affairs, American Red Cross, Gaithersburg, MD.
²⁸ Department of Pharmaceutical Chemistry, University of California, San Francisco, San Francisco, CA 94158.
²⁹ Parker Institute for Cancer Immunotherapy, San Francisco, CA, USA.
³⁰ Chan Zuckerberg Biohub, San Francisco, CA, USA.
³¹ Institute for Human Genetics, University of California, San Francisco, San Francisco, CA, USA.
³² Division of Rheumatology, Department of Medicine, University of California, San Francisco, San Francisco, CA, USA.
³³ Institute of Computational Health Sciences, University of California, San Francisco, San Francisco, CA, USA.
³⁴ Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, CA, USA.
³⁵ Department of Cellular and Molecular Pharmacology, University of California, San Francisco, CA 94158, USA.
³⁶ Quantitative Biosciences Institute, University of California, San Francisco, CA 94158, USA.

PMID: 32511497
PMCID: PMC7273265
DOI: 10.1101/2020.04.25.20074856

Test performance evaluation of SARS-CoV-2 serological assays

Jeffrey D Whitman et al. medRxiv. 2020.

[Preprint]. 2020 May 17:2020.04.25.20074856.

doi: 10.1101/2020.04.25.20074856.

Authors

Affiliations

¹ Department of Laboratory Medicine, University of California, San Francisco, San Francisco, CA 94143, USA.
² Medical Scientist Training Program, University of California, San Francisco, CA 94143, USA.
³ Biomedical Sciences Graduate Program, University of California, San Francisco, CA 94143, USA.
⁴ J. David Gladstone Institutes, San Francisco, CA 94158, USA.
⁵ Department of Microbiology and Immunology, University of California, San Francisco, CA 94143, USA.
⁶ Innovative Genomics Institute, University of California, Berkeley, CA 94720, USA.
⁷ Diabetes Center, University of California, San Francisco, San Francisco, CA 94143, USA.
⁸ Howard Hughes Medical Institute, University of California, San Francisco.
⁹ Department of Pathology, Massachusetts General Hospital/Harvard Medical School, Boston, MA, USA.
¹⁰ Department of Medicine, Division of Infectious Diseases, University of California, San Francisco, San Francisco, CA 94143, USA.
¹¹ Department of Bioengineering, University of California, Berkeley, Berkeley CA 94720 USA.
¹² Department of Integrative Biology, University of California, Berkeley, Berkeley CA 94720 USA.
¹³ Department of Surgery, University of California, San Francisco, CA 94143, USA.
¹⁴ Department of Medicine, University of California, San Francisco, San Francisco, CA 94143, USA.
¹⁵ Department of Neurology, University of California, San Francisco, CA 94158, USA.
¹⁶ Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco, San Francisco, CA 94158, USA.
¹⁷ Department of Dermatology, University of California, San Francisco, San Francisco, CA 94158, USA.
¹⁸ Program in Quantitative Biology, University of California, San Francisco, San Francisco, CA 94158, USA.
¹⁹ School of Medicine, University of California, San Francisco, San Francisco, CA 94158, USA.
²⁰ Department of Pathology, University of California, San Francisco, San Francisco, CA, 94158, USA.
²¹ Division of Experimental Medicine, Department of Medicine, University of California San Francisco, San Francisco CA, USA.
²² Infectious Diseases and Immunity Graduate Group, University of California Berkeley, Berkeley, CA, USA.
²³ Weill Institute for Neurosciences, Department of Neurology, University of California, San Francisco, San Francisco, CA.
²⁴ Department of Biochemistry and Biophysics, University of California, San Francisco, San Francisco, CA 94158, USA.
²⁵ Division of Infectious Diseases, Massachusetts General Hospital/Harvard Medical School, Boston, MA, USA.
²⁶ UCSF-Abbott Viral Diagnostics and Discovery Center, San Francisco, CA, USA.
²⁷ Scientific Affairs, American Red Cross, Gaithersburg, MD.
²⁸ Department of Pharmaceutical Chemistry, University of California, San Francisco, San Francisco, CA 94158.
²⁹ Parker Institute for Cancer Immunotherapy, San Francisco, CA, USA.
³⁰ Chan Zuckerberg Biohub, San Francisco, CA, USA.
³¹ Institute for Human Genetics, University of California, San Francisco, San Francisco, CA, USA.
³² Division of Rheumatology, Department of Medicine, University of California, San Francisco, San Francisco, CA, USA.
³³ Institute of Computational Health Sciences, University of California, San Francisco, San Francisco, CA, USA.
³⁴ Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, CA, USA.
³⁵ Department of Cellular and Molecular Pharmacology, University of California, San Francisco, CA 94158, USA.
³⁶ Quantitative Biosciences Institute, University of California, San Francisco, CA 94158, USA.

PMID: 32511497
PMCID: PMC7273265
DOI: 10.1101/2020.04.25.20074856

Update in

Evaluation of SARS-CoV-2 serology assays reveals a range of test performance.
Whitman JD, Hiatt J, Mowery CT, Shy BR, Yu R, Yamamoto TN, Rathore U, Goldgof GM, Whitty C, Woo JM, Gallman AE, Miller TE, Levine AG, Nguyen DN, Bapat SP, Balcerek J, Bylsma SA, Lyons AM, Li S, Wong AW, Gillis-Buck EM, Steinhart ZB, Lee Y, Apathy R, Lipke MJ, Smith JA, Zheng T, Boothby IC, Isaza E, Chan J, Acenas DD 2nd, Lee J, Macrae TA, Kyaw TS, Wu D, Ng DL, Gu W, York VA, Eskandarian HA, Callaway PC, Warrier L, Moreno ME, Levan J, Torres L, Farrington LA, Loudermilk RP, Koshal K, Zorn KC, Garcia-Beltran WF, Yang D, Astudillo MG, Bernstein BE, Gelfand JA, Ryan ET, Charles RC, Iafrate AJ, Lennerz JK, Miller S, Chiu CY, Stramer SL, Wilson MR, Manglik A, Ye CJ, Krogan NJ, Anderson MS, Cyster JG, Ernst JD, Wu AHB, Lynch KL, Bern C, Hsu PD, Marson A. Whitman JD, et al. Nat Biotechnol. 2020 Oct;38(10):1174-1183. doi: 10.1038/s41587-020-0659-0. Epub 2020 Aug 27. Nat Biotechnol. 2020. PMID: 32855547 Free PMC article.

Abstract

Background: Serological tests are crucial tools for assessments of SARS-CoV-2 exposure, infection and potential immunity. Their appropriate use and interpretation require accurate assay performance data.

Method: We conducted an evaluation of 10 lateral flow assays (LFAs) and two ELISAs to detect anti-SARS-CoV-2 antibodies. The specimen set comprised 128 plasma or serum samples from 79 symptomatic SARS-CoV-2 RT-PCR-positive individuals; 108 pre-COVID-19 negative controls; and 52 recent samples from individuals who underwent respiratory viral testing but were not diagnosed with Coronavirus Disease 2019 (COVID-19). Samples were blinded and LFA results were interpreted by two independent readers, using a standardized intensity scoring system.

Results: Among specimens from SARS-CoV-2 RT-PCR-positive individuals, the percent seropositive increased with time interval, peaking at 81.8-100.0% in samples taken >20 days after symptom onset. Test specificity ranged from 84.3-100.0% in pre-COVID-19 specimens. Specificity was higher when weak LFA bands were considered negative, but this decreased sensitivity. IgM detection was more variable than IgG, and detection was highest when IgM and IgG results were combined. Agreement between ELISAs and LFAs ranged from 75.7-94.8%. No consistent cross-reactivity was observed.

Conclusion: Our evaluation showed heterogeneous assay performance. Reader training is key to reliable LFA performance, and can be tailored for survey goals. Informed use of serology will require evaluations covering the full spectrum of SARS-CoV-2 infections, from asymptomatic and mild infection to severe disease, and later convalescence. Well-designed studies to elucidate the mechanisms and serological correlates of protective immunity will be crucial to guide rational clinical and public health policies.

PubMed Disclaimer

Conflict of interest statement

Competing Interests This work was supported by gifts from Anthem Blue Cross Blue Shield, the Chan Zuckerberg Biohub, and anonymous philanthropy. C.Y.C. is the director of the UCSF-Abbott Viral Diagnostics and Discovery Center, receives research support funding from Abbott Laboratories and is on the Scientific Advisory Board of Mammoth Biosciences, Inc. C. J. Y. is cofounder of DropPrint Genomics and serves as an advisor to them. M.S.A. holds stock in Medtronic and Merck. P.D.H. is a cofounder of Spotlight Therapeutics and serves on the board of directors and scientific advisory board, and is an advisor to Serotiny. P.D.H. holds stock in Spotlight Therapeutics and Editas Medicine. A.M. is a cofounder of Spotlight Therapeutics and Arsenal Biosciences and serves on their boards of directors and scientific advisory boards. A.M. has served as an advisor to Juno Therapeutics, was a member of the scientific advisory board at PACT Pharma, and was an advisor to Trizell. A.M. owns stock in Arsenal Biosciences, Spotlight Therapeutics and PACT Pharma. RY owns stock in Abbvie, Bluebird Bio, Bristol Myers Squibb, Cara Therapeutics, Editas Medicine, Esperion, and Gilead Sciences. Unrelated to this current work, the Marson lab has received sponsored research support from Juno Therapeutics, Epinomics and Sanofi, and a gift from Gilead.

Figures

**Figure 1:**
Performance data for immunochromatographic lateral flow assays (LFAs). A. The second reader’s score (0–6 based on band intensity) is reported for each assay, binned by time after patient-reported symptom onset. For tests with separate IgG and IgM bands, the higher score is reported. Joint IgM/IgG signal is represented by a single band in Wondfo. The lower, dark grey line refers to the positivity threshold (Score greater than or equal to 1) used in this study. The upper, light grey line refers to an alternative positivity threshold (Score greater than or equal to 2) discussed in the text and eFigure 4. B. Percent of SARS-CoV-2 RT-PCR-positive samples testing positive by each LFA are plotted relative to time after patient-reported symptom onset. The “IgM or IgG” category refers to positivity of either isotype. C. Specificity is plotted for each test using pre-COVID-19 negative control samples. All error bars signify 95% confidence intervals.

**Figure 2:**
LFA and ELISA values by serological assay. A. LFA scores for each of two readers (blue) and mean ELISA Signal/Cutoff Ratio (S/CO, purple) for each specimen are grouped by binned time after patient-reported symptom onset and plotted by assay. White cells indicate samples not run with the corresponding assay. For ELISAs, grey indicates S/CO less than or equal to 1. The same legend applies to Panels B and C. The F(ab’)2 specific secondary antibody used in our in-house ELISA preferentially binds the IgG light chain but has some reactivity for other isotypes (IgM, IgA). B. LFA score and ELISA S/CO values are plotted for pre-COVID-19 historical control serum samples to determine assay specificity. C. LFA score and ELISA S/CO values are plotted for serum samples obtained from 52 individuals after the emergence of COVID-19 (post-COVID-19), some of which received Biofire FilmArray (BioFire Diagnostics, Salt Lake City, UT) and/or SARS-CoV-2 RT-PCR testing (all negative) as indicated (black cells) in the appropriate columns. Arrows highlight specimens from five individuals with moderate to strong band intensity further discussed in the text. Specimens are grouped by positive testing for Coronavirus HKU1 (CoV HKU1), Coronavirus OC43 (CoV OC43), Influenza A Virus A/H3 (FluA H3), Influenza A Virus A/H1 2009 (FluA H1), Parainfluenza Type 1 Virus (PIV-1), Parainfluenza Type 4 Virus (PIV-4), Human Metapneumovirus (HMP), Adenovirus (ADNV), Respiratory Syncytial Virus (RSV), Human Rhinovirus/Enterovirus (HRE), or negative testing for SARS-CoV-2 and other viruses (nco-).

**Figure 3:**
Agreement of serological assays for SARS-CoV-2. A. Percent agreement is plotted across all assay combinations, and values signify the binomial regression of the two assays across all tests. Samples were labeled “positive” if any one isotype was detected (LFA score ≥ 1, S/CO > 1) for each assay. B. IgM or IgG LFA scores for each assay are compared to Signal/Cutoff Ratios (S/CO) from three different ELISAs for all SARS-CoV-2 RT-PCR-positive samples. Joint IgM/IgG signal is represented by a single band in Wondfo, so data were plotted as IgM or IgG depending on ELISA comparison. The F(ab’)2 specific secondary antibody used in our in-house ELISA preferentially binds the IgG light chain but contains some reactivity for other isotypes (IgM, IgA). Error bars signify 95% confidence intervals.

See this image and copyright information in PMC

Cited by

Serological profiles of pan-coronavirus-specific responses in COVID-19 patients using a multiplexed electro-chemiluminescence-based testing platform.
Chaudhury S, Hutter J, Bolton JS, Hakre S, Mose E, Wooten A, O'Connell W, Hudak J, Krebs SJ, Darden JM, Regules JA, Murray CK, Modjarrad K, Scott P, Peel S, Bergmann-Leitner ES. Chaudhury S, et al. PLoS One. 2021 Jun 3;16(6):e0252628. doi: 10.1371/journal.pone.0252628. eCollection 2021. PLoS One. 2021. PMID: 34081747 Free PMC article.
The Rapid Coronavirus Antibody Test: Can We Improve Accuracy?
Pavlova IP, Nair SS, Kyprianou N, Tewari AK. Pavlova IP, et al. Front Med (Lausanne). 2020 Sep 2;7:569. doi: 10.3389/fmed.2020.00569. eCollection 2020. Front Med (Lausanne). 2020. PMID: 32984390 Free PMC article. No abstract available.
A scaling approach to estimate the age-dependent COVID-19 infection fatality ratio from incomplete data.
Seoane B. Seoane B. PLoS One. 2021 Feb 17;16(2):e0246831. doi: 10.1371/journal.pone.0246831. eCollection 2021. PLoS One. 2021. PMID: 33596249 Free PMC article.
Rapid point-of-care testing for COVID-19: quality of supportive information for lateral flow serology assays.
Kierkegaard P, McLister A, Buckle P. Kierkegaard P, et al. BMJ Open. 2021 Mar 19;11(3):e047163. doi: 10.1136/bmjopen-2020-047163. BMJ Open. 2021. PMID: 33741675 Free PMC article.
Diagnostic accuracy of serological tests for covid-19: systematic review and meta-analysis.
Lisboa Bastos M, Tavaziva G, Abidi SK, Campbell JR, Haraoui LP, Johnston JC, Lan Z, Law S, MacLean E, Trajman A, Menzies D, Benedetti A, Ahmad Khan F. Lisboa Bastos M, et al. BMJ. 2020 Jul 1;370:m2516. doi: 10.1136/bmj.m2516. BMJ. 2020. PMID: 32611558 Free PMC article.

See all "Cited by" articles

References

1. Johns Hopkins University. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594.... Published 2020. Accessed April 12, 2020.
1. Infectious Diseases Society of America. IDSA Statement on COVID-19 Testing. https://www.idsociety.org/globalassets/idsa/public-health/covid-19-idsa-.... Published 2020. Accessed April 20th, 2020.
1. Weitz JS, Beckett SJ, Coenen AR, et al. Modeling shield immunity to reduce COVID-19 epidemic spread. Nat Med. 2020. - PMC - PubMed
1. World Health Organization. Laboratory testing for coronavirus disease 2019 (COVID-19) in suspected human cases: interim guidance, 2 March 2020. Geneva: World Health Organization; 2020; 2020.
1. Zhao J, Yuan Q, Wang H, et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis. 2020. - PMC - PubMed

Publication types

Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Miscellaneous
- NCI CPTAC Assay Portal

[1] Johns Hopkins University. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594.... Published 2020. Accessed April 12, 2020.

[2] Johns Hopkins University. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594.... Published 2020. Accessed April 12, 2020.

[3] Infectious Diseases Society of America. IDSA Statement on COVID-19 Testing. https://www.idsociety.org/globalassets/idsa/public-health/covid-19-idsa-.... Published 2020. Accessed April 20th, 2020.

[4] Infectious Diseases Society of America. IDSA Statement on COVID-19 Testing. https://www.idsociety.org/globalassets/idsa/public-health/covid-19-idsa-.... Published 2020. Accessed April 20th, 2020.

[5] Weitz JS, Beckett SJ, Coenen AR, et al. Modeling shield immunity to reduce COVID-19 epidemic spread. Nat Med. 2020. - PMC - PubMed

[6] Weitz JS, Beckett SJ, Coenen AR, et al. Modeling shield immunity to reduce COVID-19 epidemic spread. Nat Med. 2020. - PMC - PubMed

[7] World Health Organization. Laboratory testing for coronavirus disease 2019 (COVID-19) in suspected human cases: interim guidance, 2 March 2020. Geneva: World Health Organization; 2020; 2020.

[8] World Health Organization. Laboratory testing for coronavirus disease 2019 (COVID-19) in suspected human cases: interim guidance, 2 March 2020. Geneva: World Health Organization; 2020; 2020.

[9] Zhao J, Yuan Q, Wang H, et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis. 2020. - PMC - PubMed

[10] Zhao J, Yuan Q, Wang H, et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis. 2020. - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

This is a preprint.

Test performance evaluation of SARS-CoV-2 serological assays

Affiliations

Test performance evaluation of SARS-CoV-2 serological assays

Authors

Affiliations

Update in

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

Grants and funding

LinkOut - more resources

Full Text Sources

Miscellaneous