Benchmarking of methods for genomic taxonomy

Abstract

One of the first issues that emerges when a prokaryotic organism of interest is encountered is the question of what it is--that is, which species it is. The 16S rRNA gene formed the basis of the first method for sequence-based taxonomy and has had a tremendous impact on the field of microbiology. Nevertheless, the method has been found to have a number of shortcomings. In the current study, we trained and benchmarked five methods for whole-genome sequence-based prokaryotic species identification on a common data set of complete genomes: (i) SpeciesFinder, which is based on the complete 16S rRNA gene; (ii) Reads2Type that searches for species-specific 50-mers in either the 16S rRNA gene or the gyrB gene (for the Enterobacteraceae family); (iii) the ribosomal multilocus sequence typing (rMLST) method that samples up to 53 ribosomal genes; (iv) TaxonomyFinder, which is based on species-specific functional protein domain profiles; and finally (v) KmerFinder, which examines the number of cooccurring k-mers (substrings of k nucleotides in DNA sequence data). The performances of the methods were subsequently evaluated on three data sets of short sequence reads or draft genomes from public databases. In total, the evaluation sets constituted sequence data from more than 11,000 isolates covering 159 genera and 243 species. Our results indicate that methods that sample only chromosomal, core genes have difficulties in distinguishing closely related species which only recently diverged. The KmerFinder method had the overall highest accuracy and correctly identified from 93% to 97% of the isolates in the evaluations sets.

FIG 1

Performance of the five methods for species identification on the indicated data sets. The rMLST and TaxonomyFinder methods take only draft or complete genomes as input, while Reads2Type works only for short reads. Correct (genus and species), predicted genus and species are in accordance with the annotation; only genus correct, the predicted genus is in accordance with the annotation, but the species is not; not even genus correct, neither predicted genus nor species is in accordance with the annotation.

FIG 2

Overlap in predictions by the five methods for species identification. Numbers written in regular font indicate the number of isolates for which the predicted species corresponds to the annotated species. Numbers written in italics indicate the number of isolates for which the predicted and annotated species differ. The methods used and data sets evaluated are indicated.

FIG 3

Predictions for the most common species of the NCBI_drafts set. For each method, indicated at the top of each panel, the results for a given species are only shown if the method made a prediction for five or more isolates annotated as this species (e.g., if there are five isolates annotated as species A in the data set, but the method was not able to make a prediction for one of the isolates, the species is not shown) or if two or more isolates are predicted as this species (e.g., if there are no isolates annotated as species B in the data set but two isolates annotated as species C are predicted to be species B, then species B is shown).

FIG 4

Predictions for the most common species in the SRA_drafts data set. For each method, indicated at the top of each panel, the results for a given species is shown only if the method made a prediction for 10 or more isolates annotated as this species or if two or more isolates are predicted as this species.