Multicenter assessment of shotgun metagenomics for pathogen detection

Abstract

Background: Shotgun metagenomics has been used clinically for diagnosing infectious diseases. However, most technical assessments have been limited to individual sets of reference standards, experimental workflows, and laboratories.

Methods: A reference panel and performance metrics were designed and used to examine the performance of shotgun metagenomics at 17 laboratories in a coordinated collaborative study. We comprehensively assessed the reliability, key performance determinants, reproducibility, and quantitative potential.

Findings: Assay performance varied significantly across sites and microbial classes, with a read depth of 20 millions as a generally cost-efficient assay setting. Results of mapped reads by shotgun metagenomics could indicate relative and intra-site (but not absolute or inter-site) microbial abundance.

Interpretation: Assay performance was significantly impacted by the microbial type, the host context, and read depth, which emphasizes the importance of these factors when designing reference reagents and benchmarking studies. Across sites, workflows and platforms, false positive reporting and considerable site/library effects were common challenges to the assay's accuracy and quantifiability. Our study also suggested that laboratory-developed shotgun metagenomics tests for pathogen detection should aim to detect microbes at 500 CFU/mL (or copies/mL) in a clinically relevant host context (10^5 human cells/mL) within a 24h turn-around time, and with an efficient read depth of 20M.

Funding: This work was supported by National Science and Technology Major Project of China (2018ZX10102001).

Keywords: Diagnostic Performance; Multicenter Assessment; Next-generation Sequencing; Shotgun Metagenomics for Pathogen Detection.

Fig. 1

Design and overview of the study. (a) Thirty microorganisms from 19 genera were chosen to represent the diversity of microbial types (left panel), genome sizes, and GC contents (right panel). (b) Pathogen reference reagents (PRHs and PRLs) were prepared by contriving microbes at high (PRH) and low (PRL) titers with HeLa cells, respectively. PRH and PRL samples were diluted 1:10 and tested in ten replicates except for the pathogen-free PRC sample. PRC details are in the Results section. (c) Overview of study design. Numbers (1-5) order the steps of analysis. F-score, Recall and Precision metrics were used to assess the performance of accuracy, sensitivity, and specificity. Key technical factors of the workflow were explored for their impact on assay performance. See also Supplemental Tables S1–S3, and S5.

Fig. 2

Assessment of assay performance across sites. (a) Summary of assay performance as measured by recall, precision, and F-score by site (n = 136). (b) Summary of assay performance by measure (n = 17). (c) Visualization of the similarity in microbial compositions across sites and reference reagents (n = 136). A nMDS plot of a Bray-Curtis dissimilarity matrix was constructed from the species composition of each reference reagent at each site (n = 136). (d) Summary of assay performance by microbial type, (n = 136), *, P < 0.05, **, P < 0.01 by Wilcoxon rank sum test. (e) Detection of each microorganism in the reference panel. Bacteria and fungi were measured in CFU/ml, and viruses were measured in copies/ml. Columns indicate the number of detection sites, and black dots indicate titers of microorganisms (n = 136). Ranges of abundances of each microbial type are displayed at the top. (f) Assay turn-around times by site (n = 17). (g) Assay turn-around times by step (n = 17). See also Supplemental Figs. S1, S2.

Fig. 3

Microbial abundance and pathogen detection by metagenomics. (a,b) Observed abundance ratios between PRH and their PRL counterparts, and undiluted samples and their diluted counterparts, analyzed together (a) or by microbial type (b). (c) Correlations between relative abundance and performance (n = 136). (d) Correlations between observed and expected abundances in three microbial types, using the reads per million (RPM) of human papilloma virus within each sample as an internal control for normalization. (e,f) RPM of microbial detection varied across sites grouped by workflow, Hd: Host-deplete, UNC: Ultrasound, None, Column; UBC: Ultrasound, Bead−beating, Column; EBC: Endonuclease, Bead−beating, Column; TND: Transposase, None, Dynabeads; TBC: Transposase, Bead−beating, Column; UND: Ultrasound, None, Dynabeads; EBD: Endonuclease, Bead−beating, Dynabeads. (e) or key technical variables (f). (n = 136), *, P < 0.05, **, P < 0.01 by Wilcoxon rank sum test. See also Supplemental Figs. S3–S5. (g) Correlations between F-score and major workflow-dependent technical variables, (n = 270). **, P < 0.01 by Wilcoxon rank sum test.

Fig. 4

Assessment of assay reproducibility of pathogen metagenomics. (a–c) Coefficients of variance (CV) of microbial mapped reads across sites sorted by workflows (a), or grouped by technical variables (b), or microbial type (c). (d,e) Comparison between the overall CV (observed) and the sampling CV (simulated), with data from all sites analyzed together (d) or individually (e). (f) Linear regression with sampling simulated CV as the independent variable. **, P < 0.01 by Wilcoxon rank sum test. See also Supplemental Table S6.

Fig. 5

Associations between assay performance and major technical variables. (a) Correlations between performance metrics and sequencing quality as inferred by Q30 score. (b) Correlations between site performance metrics and read depth. (c,d) Performance of pathogen detection as measured by Recall as a function of read depth. Data were grouped and analyzed by each reference reagent (c) or microbial type (d). See also Fig. S6–S8.

Fig. 6

Reducing false-positive results in pathogen metagenomics. (a) Four main causes of false-positive results (n = 17). (b) Patterns of background microbes clustered dependent on methods of library preparation and nucleic acid extraction. (c) Summary of the prevalence of background microbes (n = 17). (d) Improved assay specificity after applying filters to remove false-positive results.