Evaluation of shotgun metagenomics sequence classification methods using in silico and in vitro simulated communities

Abstract

Background: The field of metagenomics (study of genetic material recovered directly from an environment) has grown rapidly, with many bioinformatics analysis methods being developed. To ensure appropriate use of such methods, robust comparative evaluation of their accuracy and features is needed. For taxonomic classification of sequence reads, such evaluation should include use of clade exclusion, which better evaluates a method's accuracy when identical sequences are not present in any reference database, as is common in metagenomic analysis. To date, relatively small evaluations have been performed, with evaluation approaches like clade exclusion limited to assessment of new methods by the authors of the given method. What is needed is a rigorous, independent comparison between multiple major methods, using the same in silico and in vitro test datasets, with and without approaches like clade exclusion, to better characterize accuracy under different conditions.

Results: An overview of the features of 38 bioinformatics methods is provided, evaluating accuracy with a focus on 11 programs that have reference databases that can be modified and therefore most robustly evaluated with clade exclusion. Taxonomic classification of sequence reads was evaluated using both in silico and in vitro mock bacterial communities. Clade exclusion was used at taxonomic levels from species to class-identifying how well methods perform in progressively more difficult scenarios. A wide range of variability was found in the sensitivity, precision, overall accuracy, and computational demand for the programs evaluated. In experiments where distilled water was spiked with only 11 bacterial species, frequently dozens to hundreds of species were falsely predicted by the most popular programs. The different features of each method (forces predictions or not, etc.) are summarized, and additional analysis considerations discussed.

Conclusions: The accuracy of shotgun metagenomics classification methods varies widely. No one program clearly outperformed others in all evaluation scenarios; rather, the results illustrate the strengths of different methods for different purposes. Researchers must appreciate method differences, choosing the program best suited for their particular analysis to avoid very misleading results. Use of standardized datasets for method comparisons is encouraged, as is use of mock microbial community controls suitable for a particular metagenomic analysis.

Fig. 1

Performance as clade exclusion level is varied. Sensitivity (a) and precision (b) on the MetaSimHC dataset of simulated 250 bp reads. There is a wide range of variability in the sensitivity and precision of the methods with sensitivity tending to decrease as the level of clade exclusion moves from species to class. Performance is calculated based on proportion of reads appropriately assigned and averaged per genome (see Methods)

Fig. 2

Distributions of assignments to taxonomic ranks. Proportion of reads assigned at each taxonomic rank on the MetaSimHC dataset of simulated 250 bp reads under genus clade exclusion (includes both correct and incorrect assignments). Although the lowest possible correct rank is family, many methods still classify the majority of reads at the species level. CARMA3 and DiScRIBinATE are slightly more conservative, classifying a large number of reads at the family or order levels, whereas TACOA is extremely conservative, classifying the majority of the reads at the superkingdom level

Fig. 3

Performance as clade exclusion level is varied with overpredictions (see Methods for details) classified as correct. Sensitivity (a) and precision (b) on the MetaSimHC dataset of simulated 250 bp reads. Methods such as MEGAN4 which classify many reads at lower taxonomic levels see a considerable increase in performance, whereas more conservative methods such as CARMA3 see only a slight improvement. Performance is calculated based on proportion of reads appropriately assigned and averaged per genome (see Methods)

Fig. 4

Performance of FW in silico versus FW in vitro. Sensitivity (a) and precision (b) of methods on the FW dataset comparing the performance on the in silico community versus the in vitro community under species clade exclusion. The results are similar between the in vitro and in silico communities, demonstrating that methods appear to be relatively robust to real Illumina sequencing errors for this simple community. Performance is calculated based on proportion of reads appropriately assigned and averaged per genome (see Methods)

Fig. 5

Performance of MetaSimHC compared to FW in silico. Sensitivity (a) and precision (b) of methods on the MetaSimHC dataset compared to the FW in silico of simulated 250 bp reads. Values are averaged over all levels of clade exclusion from species to class. Although the microbes in the dataset changed, the relative performance of the methods remains very similar. Performance is calculated based on proportion of reads appropriately assigned and averaged per genome (see Methods)