Benchmarking metagenomic binning tools on real datasets across sequencing platforms and binning modes

Abstract

Metagenomic binning is a culture-free approach that facilitates the recovery of metagenome-assembled genomes by grouping genomic fragments. However, there remains a lack of a comprehensive benchmark to evaluate the performance of metagenomic binning tools across various combinations of data types and binning modes. In this study, we benchmark 13 metagenomic binning tools using short-read, long-read, and hybrid data under co-assembly, single-sample, and multi-sample binning, respectively. The benchmark results demonstrate that multi-sample binning exhibits optimal performance across short-read, long-read, and hybrid data. Moreover, multi-sample binning outperforms other binning modes in identifying potential antibiotic resistance gene hosts and near-complete strains containing potential biosynthetic gene clusters across diverse data types. This study also recommends three efficient binners across all data-binning combinations, as well as high-performance binners for each combination.

Fig. 1. Number of NC and HQ MAGs recovered from five real datasets.

a, c, e, g, i The number of NC MAGs recovered from marine, cheese, human gut I, human gut II and activated sludge datasets respectively. b, d, f, h, j The number of HQ MAGs recovered from marine, cheese, human gut I, human gut II and activated sludge datasets respectively. The description of seven data-binning combinations can be seen in Table 2. “nan” denotes that the corresponding binner failed to complete execution within two weeks based on the computational resources we used.

Fig. 2. Overall ranking scores (see Section Evaluation metrics and ranking score), runtime, and memory usage for each binner across seven data-binning combinations.

a Overall ranking scores of each binner across the seven data-binning combinations. The rankings of the binners, based on their overall ranking scores, were annotated. “nan” denotes that CONCOCT, MaxBin 2, MetaBinner, and COMEBin (GPU) cannot complete execution within two weeks on the activated sludge short-read co-assembly data (see Section Computational resources for computational resources). Consequently, these binners are ranked last on the activated sludge short-read co-assembly data (see Supplementary Fig. 3e). b, c Runtime and memory usage for each binner in the activated sludge dataset. According to the recommendations in the referenced study, COMEBin was used in GPU environments. In the short_co data-binning combination, there is a single assembly, with each bar representing the runtime or memory usage for each binner. In the remaining six data-binning combinations, there are 23 assemblies (N = 23) per combination, with each bar showing the average runtime or memory usage for each binner. The error bars indicate the standard deviation.

Fig. 3. Comparison of bin-refinement tool performance across seven data-binning combinations.

a, d, g, j, m The number of MQ MAGs recovered from marine, cheese, human gut I, human gut II and activated sludge datasets, respectively. b, e, h, k, n The number of NC MAGs recovered from marine, cheese, human gut I, human gut II and activated sludge datasets, respectively. c, f, i, l, o The number of HQ MAGs recovered from marine, cheese, human gut I, human gut II and activated sludge datasets, respectively.

Fig. 4. Number of species and strains recovered from the marine dataset.

a, b Number of species or strains after dereplication of refined NC MAGs across seven data-binning combinations. c, d Number of species or strains after dereplication of refined HQ MAGs across seven data-binning combinations.