Major data analysis errors invalidate cancer microbiome findings

Abstract

We re-analyzed the data from a recent large-scale study that reported strong correlations between microbial organisms and 33 different cancer types, and that created machine learning predictors with near-perfect accuracy at distinguishing among cancers. We found at least two fundamental flaws in the reported data and in the methods: (1) errors in the genome database and the associated computational methods led to millions of false positive findings of bacterial reads across all samples, largely because most of the sequences identified as bacteria were instead human; and (2) errors in transformation of the raw data created an artificial signature, even for microbes with no reads detected, tagging each tumor type with a distinct signal that the machine learning programs then used to create an apparently accurate classifier. Each of these problems invalidates the results, leading to the conclusion that the microbiome-based classifiers for identifying cancer presented in the study are entirely wrong. These flaws have subsequently affected more than a dozen additional published studies that used the same data and whose results are likely invalid as well.

Figure 1.

Average number of reads per sample in bladder cancer (BLCA) in the top 20 most-abundant genera reported in Poore et al. (left), averaged across 156 whole-genome sequencing samples. On the right are the counts for the same samples and same genera, in the same order, as computed in our re-analysis. Note that the y-axis scales are different by a factor of 2000. The x-axis shows genus names.

Figure 2.

Distribution of normalized counts for Hepandensovirus for Adrenocortical carcinoma (blue) versus all other samples (orange). Inset shows zoomed-in view of the distribution for the smallest values. All raw values were zero.

Figure 3.

Distribution of normalized counts for Thiorhodospira reads in kidney chromophobe (KICH) cancer (blue) and normal (orange) samples. Nearly all raw values were zero except for 7 samples with a raw count of 1.

Figure 4.

Distribution of normalized read counts in the APCR data set for Nitrospira reads found in lung squamous cell carcinoma (blue) and all other cancer types (orange). For clarity, the y-axis is truncated at 500, but the peak of the distribution for other cancers (organe) is at 1389.

Figure 5.

Distribution of normalized counts for Mulikevirus reads in head and neck squamous cell (HNSC) cancer (orange) and normal (blue) samples. All raw values were zero.

Figure 6.

Accuracies for one-vs-all tumor classification models obtained from a selection of samples and genera with zero classified reads prior to normalization. Each row shows the accuracies of a classifier that distinguished one cancer type from all other cancer types in the table. AUC: maximum measured area under the sensitivity-specificity curve. PPV: positive predictive value. NPV: negative predictive value.