Developing standards for the microbiome field

Abstract

Background: Effective standardisation of methodologies to analyse the microbiome is essential to the entire microbiome community. Despite the microbiome field being established for over a decade, there are no accredited or certified reference materials available to the wider community. In this study, we describe the development of the first reference reagents produced by the National Institute for Biological Standards and Control (NIBSC) for microbiome analysis by next-generation sequencing. These can act as global working standards and will be evaluated as candidate World Health Organization International Reference Reagents.

Results: We developed the NIBSC DNA reference reagents Gut-Mix-RR and Gut-HiLo-RR and a four-measure framework for evaluation of bioinformatics tool and pipeline bias. Using these reagents and reporting system, we performed an independent evaluation of a variety of bioinformatics tools by analysing shotgun sequencing and 16S rRNA sequencing data generated from the Gut-Mix-RR and Gut-HiLo-RR. We demonstrate that key measures of microbiome health, such as diversity estimates, are largely inflated by the majority of bioinformatics tools. Across all tested tools, biases were present, with a clear trade-off occurring between sensitivity and the relative abundance of false positives in the final dataset. Using commercially available mock communities, we investigated how the composition of reference reagents may impact benchmarking studies. Reporting measures consistently changed when the same bioinformatics tools were used on different community compositions. This was influenced by both community complexity and taxonomy of species present. Both NIBSC reference reagents, which consisted of gut commensal species, proved to be the most challenging for the majority of bioinformatics tools tested. Going forward, we recommend the field uses site-specific reagents of a high complexity to ensure pipeline benchmarking is fit for purpose.

Conclusions: If a consensus of acceptable levels of error can be agreed on, widespread adoption of these reference reagents will standardise downstream gut microbiome analyses. We propose to do this through a large open-invite collaborative study for multiple laboratories in 2020. Video Abstract.

Fig. 1

A comparison of different bioinformatics tool performances at both the species level and genera level using the NIBSC Gut-Mix-RR and Gut-HiLo-RRs. a Relative abundance of each species in the Gut-Mix-RR as calculated by five different metagenomic taxonomic profiling tools in comparison to the known composition of the reagent. b Relative abundance of each species in the Gut-HiLo-RR as calculated by five different metagenomic taxonomic profiling tools in comparison to the known composition of the reagent. c Relative abundance of each genera in the Gut-Mix-RR as calculated by five different metagenomic taxonomic profiling tools and two 16S rRNA taxonomic profiling pipelines in comparison to the known composition of the reagent. d Relative abundance of each genera in the Gut-HiLo-RR as calculated by five different metagenomic taxonomic profiling tools and two 16S rRNA taxonomic profiling pipelines in comparison to the known composition of the reagent. e Reporting measures for pipeline performance for calculating species as evaluated using the Gut-Mix-RR. f Reporting measures for pipeline performance for calculating species as evaluated using the Gut-HiLo-RR. g Reporting measures for pipeline performance for calculating genera as evaluated using the Gut-Mix-RR. h Reporting measures for pipeline performance for calculating genera as evaluated using the Gut-HiLo-RR. Mp MetaPhlAn2, Kj Kaiju, Kr Kraken, Br Bracken, Cn Centrifuge, Sens sensitivity, FPRA false positive relative abundance, Div diversity, Sim similarity

Fig. 2

Visualisation of the relationship between different mock communities following sequencing and taxonomic profiling by a variety of approaches. A nMDS plot of a Bray-Curtis dissimilarity matrix was constructed from the species composition of five reference reagents following shotgun sequencing and taxonomic profiling by five different bioinformatics tools, MetaPhlAn2, Kaiju, Kraken, Bracken, and Centrifuge. Gut-HiLo = NIBSC Gut-HiLo-RR. Gut-Mix = NIBSC Gut-Mix-RR. MSA_1000 = ATCC MSA-1000. MSA_1001 = ATCC MSA-1001. MSA_1002 = ATCC MSA-1002. MSA_1003 = ATCC MSA-1003. Zymo = ZymoBIOMICS Microbial Community Standard

Fig. 3

Changes in pipeline performance for sensitivity (a) and similarity (b) when using different reference reagents to benchmark bioinformatics tool performance. Gut-HiLo = NIBSC Gut-HiLo-RR. Gut-Mix = NIBSC Gut-Mix-RR. MSA_1000 = ATCC MSA-1000. MSA_1001 = ATCC MSA-1001. MSA_1002 = ATCC MSA-1002. MSA_1003 = ATCC MSA-1003. Zymo = ZymoBIOMICS Microbial Community Standard