Proficiency Testing of Metagenomics-Based Detection of Food-Borne Pathogens Using a Complex Artificial Sequencing Dataset

Abstract

Metagenomics-based high-throughput sequencing (HTS) enables comprehensive detection of all species comprised in a sample with a single assay and is becoming a standard method for outbreak investigation. However, unlike real-time PCR or serological assays, HTS datasets generated for pathogen detection do not easily provide yes/no answers. Rather, results of the taxonomic read assignment need to be assessed by trained personnel to gain information thereof. Proficiency tests are important instruments of validation, harmonization, and standardization. Within the European Union funded project COMPARE [COllaborative Management Platform for detection and Analyses of (Re-) emerging and foodborne outbreaks in Europe], we conducted a proficiency test to scrutinize the ability to assess diagnostic metagenomics data. An artificial dataset resembling shotgun sequencing of RNA from a sample of contaminated trout was provided to 12 participants with the request to provide a table with per-read taxonomic assignments at species level and a report with a summary and assessment of their findings, considering different categories like pathogen, background, or contaminations. Analysis of the read assignment tables showed that the software used reliably classified the reads taxonomically overall. However, usage of incomplete reference databases or inappropriate data pre-processing caused difficulties. From the combination of the participants' reports with their read assignments, we conclude that, although most species were detected, a number of important taxa were not or not correctly categorized. This implies that knowledge of and awareness for potentially dangerous species and contaminations need to be improved, hence, capacity building for the interpretation of diagnostic metagenomics datasets is necessary.

Keywords: background contamination; diagnostic assessment; high-throughput sequencing; metagenomics; pathogen; proficiency test; training.

FIGURE 1

Summary of the read assignments at the superkingdom-level for 11 participants that provided the requested read assignment table with the read to species assignments (except P10). Only five read assignment tables (P1, P3, P4, P8, and P11) contained an assignment for all reads of the dataset. Only the compositions reported by P4, P8, and P11 fit the known actual composition (actual; upper left) of the dataset.

FIGURE 2

Positive predictive values of read assignments calculated from the complete read set, calculated based on the species assignments. Sequences of the taxa labeled gray (Brugia malayi, Caenorhabditis remanei, Danio rerio, and Scomber japonicus) were downloaded unintentionally as part of the Anisakis sequence dataset.

FIGURE 3

Sensitivity of read assignments calculated from the complete read set. (A) Sensitivities calculated based on the species assignments. (B) Sensitivities calculated based on the genus assignments. Sequences of the taxa labeled gray (Brugia malayi, Caenorhabditis remanei, Danio rerio, and Scomber japonicus) were downloaded unintentionally as part of the Anisakis sequence dataset.

FIGURE 4

Summary of the read-assignment tables (A), and assessment and interpretation of the assignments (B). Heatmap showing the positive (comprised species detected; green) and negative (comprised species NOT detected; red) results of the software analyses. The results shown for participants are based on their uploaded read assignment tables, except for P10, for which the results are derived from their summary table and assessment (A). Heatmap summarizing assessments of the detected species by the participants (B). Sequences of the taxa labeled gray (Brugia malayi, Caenorhabditis remanei, Danio rerio, and Scomber japonicus) were downloaded unintentionally as part of the Anisakis sequence dataset.