Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology

Abstract

As part of the Reproducibility Project: Cancer Biology, we published Registered Reports that described how we intended to replicate selected experiments from 29 high-impact preclinical cancer biology papers published between 2010 and 2012. Replication experiments were completed and Replication Studies reporting the results were submitted for 18 papers, of which 17 were accepted and published by eLife with the rejected paper posted as a preprint. Here, we report the status and outcomes obtained for the remaining 11 papers. Four papers initiated experimental work but were stopped without any experimental outcomes. Two papers resulted in incomplete outcomes due to unanticipated challenges when conducting the experiments. For the remaining five papers only some of the experiments were completed with the other experiments incomplete due to mundane technical or unanticipated methodological challenges. The experiments from these papers, along with the other experiments attempted as part of the Reproducibility Project: Cancer Biology, provides evidence about the challenges of repeating preclinical cancer biology experiments and the replicability of the completed experiments.

Keywords: cancer biology; human; methodology; mouse; null results; open science; replication; reproducibility; transparency.

Figure 1.. Replication attempt of Sharma et al., 2010.

Growth characteristics of PC9 cells treated with various doses of erlotinib. (A) PC9 cells were plated at equal density and treated with dimethyl sulfoxide(DMSO) or the indicated dose of erlotinib (2 or 20 µM) for 9 days (fresh erlotinib was added every 3 days). Representative microscopic images of conditions on day 9. (B) Percent of drug-tolerant persisters (DTPs) as a percentage of DMSO control cells. PC9 cells were plated at various densities (cells/cm²) and treated with DMSO or the indicated concentrations of erlotinib. (C) Survival curve of PC9 cells treated with various doses of erlotinib for 72 hr. Percent survival is relative to DMSO-treated cells. The dashed line corresponds to 50% cell killing (absolute IC₅₀ = 0.15 µM). Data plotted from one independent biological repeat. (D) Representative Western blots of lysates from PC9 cells treated with increasing concentrations of erlotinib (0.01, 0.1, 1, and 10 µM) or DMSO (0.01%). Membranes were probed with phospho-EGFR (pEGFR)-specific antibodies. Glyceraldehyde 3-phosphate dehydrogenase (GAPDH) served as loading control. Relative expression of pEGFR (to DMSO) for each concentration (0.01, 0.1, 1, and 10 µM erlotinib) is: 1.1, 1.2, 0.8, and 0.2, respectively. Each experiment was performed independently twice. Additional details can be found at https://osf.io/xbign/.

Figure 1—figure supplement 1.. Growth characteristics of PC9 cells treated with erlotinib hydrochloride (HCl).

(A) PC9 cells were plated at equal density and treated with DMSO or the indicated dose of erlotinib HCl (2 or 20 µM) for 9 days (fresh erlotinib was added every 3 days). Representative microscopic images of conditions on day 9. (B) Percent of drug-tolerant persisters (DTPs) as a percentage of DMSO control cells. PC9 cells were plated at various densities (cells/cm²) and treated with DMSO or the indicated concentrations of erlotinib HCl. (C) Survival curve of PC9 cells treated with various doses of erlotinib for 72 hr. Percent survival is relative to DMSO-treated cells. The dashed line corresponds to 50% cell killing (absolute IC₅₀ = 0.15 µM). Each experiment was performed once. Additional details can be found at https://osf.io/xbign/.

Figure 2.. Replication attempt of Ricci-Vitiani et al., 2010.

(A) Tie2 expression in patient-derived glioblastoma neurospheres (GSC83), a glioblastoma cell line (U87MG), and an endothelial cell line (human dermal microvascular endothelial cells, HMVEC). Real-time quantitative reverse transcription PCR (qRT-PCR) was performed to detect Tie2 and 18S rRNA expression. Relative expression (Tie2/18S rRNA) is presented for each cell type. Means reported and error bars represent SD from two (GSC83) or three (U87MG and HMVEC) independent biological repeats. One-way analysis of variance (ANOVA) of all groups: F(2, 5) = 5.90, p = 0.0484. Planned contrasts between HMVEC and GSC83: t(5) = 2.78, p = 0.040; U87MG and HMVEC: t(5) = 3.06, p = 0.028; GSC83 and U87MG: t(5) = 0.017, p = 0.987. (B) Meta-analysis of each effect. Effect size and 95 % confidence intervals are presented for Ricci-Vitiani et al., 2010, this replication study (Reproducibility Project: Cancer Biology, RP:CB), and a random-effects meta-analysis of those two effects. Cohen’s d is a standardized difference between the two indicated measurements where a larger value indicates a difference in relative Tie2 expression between the two cell types. Random-effects meta-analysis: HMVEC and GSC83: p = 0.531; U87MG and HMVEC: p = 0.532; GSC83 and U87MG: p = 0.695. Sample sizes used in Ricci-Vitiani et al., 2010 and RP:CB are reported under the study name. Additional details can be found at https://osf.io/mpyvx/.

Figure 3.. Replication of Kan et al., 2010.

(A) Anchorage-independent growth of human mammary epithelial cells (HMECs). HMECs were left uninfected or stably expressing empty vector (EV), wild-type (WT) GNAO1, or GNAO1-R243H. Number of colonies formed after 3 weeks. PC9 cells treated with various doses of erlotinib. Means reported and error bars represent SEM from three independent biological repeats. Student’s t test: t(4) = 5.29, p = 0.0061. (B) Representative Western blots of lysates from HMEC cells expressing the indicated conditions. Membranes were probed with FLAG-specific antibodies to detect FLAG-tagged GNAO1. B-ACTIN served as loading control. (C) Meta-analysis of each effect. Effect size and 95 % confidence intervals are presented for Kan et al., 2010, this replication study (Reproducibility Project: Cancer Biology, RP:CB), and a random-effects meta-analysis of those two effects. Cohen’s d is a standardized difference between the two indicated measurements where a larger value indicates a difference in the number of colonies between the two conditions. Random-effects meta-analysis: p = 0.016. Sample sizes used in Kan et al., 2010 and RP:CB are reported under the study name. Additional details can be found at https://osf.io/jpeqg/.

Figure 4.. Replication attempt of Heidorn et al., 2010.

ERK activation following treatment with BRAF inhibitors in A375 cells (mutant BRAF; wild-type NRAS) and D04 cells (wild-type BRAF; mutant NRAS). (A) A375 cells were treated with DMSO, sorafenib (10 µM), SB590885 (1 µM), or PD184352 (1 µM), or left untreated. Cells were harvested 4 hr later for Western blot analysis. Relative pERK1/2 expression are presented for each condition. Western blot bands were quantified, pERK1/2 was normalized to total ERK1/2, then for each biological repeat value was normalized to the untreated condition with expression presented relative to DMSO. Dot plot of independent biological repeats (n = 3). Data reported in Figure 1A of Heidorn et al., 2010 displayed as a single point (red triangle) for comparison. Planned comparison (two-tailed Wilcoxon–Mann–Whitney test) between DMSO and all other conditions: U = 2.40, uncorrected p = 0.0091, Bonferroni corrected p = 0.027, Cliff’s delta = 0.93, 95% confidence interval (CI) [0.63, 0.99]. (B) D04 cells were treated like in A and presented in the same way. Graph is separated at a dashed line to accommodate a value higher than the others. Data reported in Figure 1A of Heidorn et al., 2010 displayed as a single point (red triangle) for comparison. Planned comparisons (two-tailed Wilcoxon–Mann–Whitney tests) between DMSO and sorafenib and PD184352: U = 2.36, uncorrected p = 0.024, Bonferroni corrected p = 0.071, Cliff’s delta = 1.00, 95% CI [0.75, 1.00]; between DMSO and SB590885: U = 1.96, uncorrected p = 0.10, Bonferroni corrected p = 0.30, Cliff’s delta = 0.11, 95% CI [−0.71, 80]. (C) Representative Western blots probed with pERK1/2 (T202/Y204)-specific antibodies. Total ERK1/2 served as loading control. Additional details can be found at https://osf.io/b1aw6/.

Figure 5.. Replication attempt of Johannessen et al., 2010.

(A) Cellular dose–response curves for RAF inhibitor (PLX4720) in BRAF(V600E) cell lines, A375, HT29, and RPMI-7951. Absolute half-maximum growth inhibition (GI₅₀) values (µM) were determined for each biological repeat. GI₅₀ values unable to be accurately estimated are reported as either >10 µM, which was the highest dose tested. Data reported in Figure 3D of Johannessen et al., 2010 displayed as a single point (red triangle) for comparison. Where possible the mean and 95 % confidence interval (CI) of the replication data are shown. Planned comparison (Student’s t-test) between A375 and HT29 GI₅₀ values: t(4) = 5.90, p = 0.0041; Cohen’s d = 4.82, 95% CI [1.21, 8.36]. (B) Quantification of Western blots of lysates from RPMI-7951 cells treated with DMSO or the indicated doses of MAP3K8 kinase inhibitor for 1 hr. Membranes were probed with phospho-MEK- (pMEK), total ERK- (ERK), phospho-ERK- (pERK), or total MEK (MEK)-specific antibodies. pMEK levels were normalized to ERK, and pERK levels were normalized to MEK, and then to DMSO for each biological repeat [n = 8]. Note: Normalization of pMEK to ERK and pERK to MEK was a recommendation made by reviewers of the Registered Report (Sharma et al., 2016b). Box and whisker plots with median represented as the line through the box and whiskers representing values within 1.5 interquartile range (IQR) of the first and third quartiles. Data reported in Figure 3I of Johannessen et al., 2010 is displayed as a single point (red triangle) for comparison. One-way analysis of variance (ANOVA) on pMEK/ERK data: F(3,28) = 1.93, uncorrected p = 0.147, Bonferroni corrected p = 0.295; pERK/MEK data: F(3,28) = 1.04, uncorrected p = 0.392, Bonferroni corrected p = 0.784. Planned one-sample t-test between 20 µM and a constant of 1 (DMSO-treated cells) on pMEK/ERK data: t(7) = 2.83, uncorrected p = 0.025, Bonferroni corrected p = 0.051; Cohen’s d = −1.00, 95% CI [−1.84, −0.12]; pERK/MEK data: t(7) = 2.65, uncorrected p = 0.033, Bonferroni corrected p = 0.066; Cohen’s d = −0.94, 95% CI [−1.76, −0.07]. Additional details can be found at https://osf.io/lmhjg/.

Figure 5—figure supplement 1.. Cellular dose–response curves for each biological repeat and representative Western blot image of ERK and MEK phosphorylation.

(A) This is the same experiment as in Figure 5A. The dose–response curve of each biological repeat [n = 3] for A375, HT29, and RPMI-7951 cells treated with PLX4720 for this replication attempt are plotted. (B) This is the same experiment as in Figure 5B with the indicated doses of MAP3K8 kinase inhibitor. Representative Western blots probed with pMEK- (S217/S221), pERK- (T202/Y204), total ERK-, or total MEK-specific antibodies. Vinculin (VINC) served as loading control. Relative pMEK/ERK and pERK/MEK expressions are reported below pMEK and pERK images, respectively. Additional details can be found at https://osf.io/lmhjg/.

Figure 6.. Replication attempt of Kang et al., 2011.

(A–F) Quantification of Nras-, p21-, and p16-positive cells on liver sections from C.B-17 or C.B-17 SCID/beige mice six or 30 days after intrahepatic delivery of Nras^G12V or Nras^G12V/D38A. Percent positive cells were determined for each mouse. Five mice per group, except C.B-17 with Nras^G12V at 6 days and C.B-17 SCID/beige with Nras^G12V/D38A at 6 days, which had 6 mice per group. (G) Quantification of Nras-positive cells on liver sections from wild-type (BL/6) or CD4^−/− mice 12 days after intrahepatic delivery of Nras^G12V or Nras^G12V/D38A. Percent positive cells were determined for each mouse (n = 5 per group). Additional details can be found at https://osf.io/82nfe/.

Figure 7.. Replication attempt of Lu et al., 2012.

(A) HEK293T cells transfected with wild-type (WT) or the indicated mutant of IDH1 or IDH2, or empty vector, were analyzed for intracellular metabolites 72 hr after transfection by gas chromatography–mass spectrometry (GC–MS). Quantitation of 2HG single intensity relative to glutamate was determined using total ion chromatograms (TIC) for each biological repeat (n = 6). Box and whisker plot with median represented as the line through the box and whiskers representing values within 1.5 interquartile range (IQR) of the first and third quartiles. Means as black dot and bold error bars represent 95% CI. Data estimated from the representative experiment reported in Figure 1B of Lu et al., 2012 is displayed as a single point (red circle) for comparison. Statistical analysis was performed on log₁₀-transformed data generated during this replication attempt. One-way analysis of variance (ANOVA) of IDH1 and IDH2 groups: F(3, 20) = 56.8, p = 5.73 × 10⁻¹⁰. Planned contrasts between IDH1 WT and IDH1 R132H: t(20) = 8.33, uncorrected p = 6.23 × 10⁻⁸, Bonferroni corrected p = 1.25 × 10⁻⁷, Cohen’s d = 4.81, 95% CI [2.42, 7.15]; IDH2 WT and IDH2 R172K: t(20) = 9.35, uncorrected p = 9.71 × 10⁻⁹, Bonferroni corrected p = 1.94 × 10⁻⁸, Cohen’s d = 5.40, 95% CI [2.79, 7.96]. (B) Western blot analysis for HEK293T cells transfected with WT or the indicated mutant of IDH1 or IDH2, or empty vector. Methylation status from histone extracts of the indicated markers were normalized to H3 and then to vector for each biological repeat (n = 6). Box and whisker plot with median represented as the line through the box and whiskers representing values within 1.5 IQR of the first and third quartiles. For each marker a one-way analysis of variance (ANOVA; or Kruskal–Wallis) was performed on IDH1 and IDH2 groups and the correlation with 2HG levels in A was performed. H3K4me2: H(3) = 0.367, uncorrected p = 0.947, Bonferroni corrected p > 0.99; correlation with 2HG: t(12) = 1.12, uncorrected p = 0.275, Bonferroni corrected p > 0.99. H3K9me2: F(3,20) = 0.793, uncorrected p = 0.512, Bonferroni corrected p > 0.99; correlation with 2HG: t(12) = 1.53, uncorrected p = 0.140, Bonferroni corrected p = 0.699. H3K9me3: F(3,20) = 0.515, uncorrected p = 0.676, Bonferroni corrected p > 0.99; correlation with 2HG: t(12) = 1.35, uncorrected p = 0.191, Bonferroni corrected p = 0.957. H3K36me3: F(3,20) = 0.248, uncorrected p = 0.862, Bonferroni corrected p > 0.99; correlation with 2HG: t(12) = 0.725, uncorrected p = 0.476, Bonferroni corrected p > 0.99. H3K79me2: H(3) = 1.55, uncorrected p = 0.672, Bonferroni corrected p > 0.99; correlation with 2HG: t(12) = 0.752, uncorrected p = 0.460, Bonferroni corrected p > 0.99. Planned contrasts for H3K9me2 between IDH1 WT and IDH1 R132H: t(20) = −0.238, uncorrected p = 0.815, Bonferroni corrected p > 0.99; IDH2 WT and IDH2 R172K: t(20) = 1.31, uncorrected p = 0.203, Bonferroni corrected p > 0.99. (C) 3T3-L1 cells were transduced to express the indicated proteins and analyzed and presented in the same way as A. Number of independent biological repeats (n = 5). Data estimated from the representative experiment reported in Figure 2A of Lu et al., 2012 is displayed as a single point (red circle) for comparison. Statistical analysis was performed on log₁₀-transformed data generated during this replication attempt. Student’s t-test of IDH2 WT and IDH2 R172K: t(8) = 14.3, p = 5.49 × 10⁻⁷, Cohen’s d = 9.06, 95% CI [4.53, 13.57]. Additional details can be found at https://osf.io/vfsbo/.

Figure 7—figure supplement 1.. Replication attempt of Lu et al., 2012.

(A) This is the same experiment as in Figure 7B. Representative Western blots (experiment performed independently six times) of histone extractions from HEK293T cells transfected with the indicated plasmids. Membranes were probed with the indicated antibodies (total H3 served as loading control). (B) This is the same experiment as in Figure 7A. Representative total ion chromatograms (TIC) from samples harvested 72 hr after transfection for the indicated transfected HEK293T cells. The detection of 2HG and glutamate are shown and based on spectra of derivatized commercial standards. (C) This is the same experiment as in Figure 7C. Representative Western blots (experiment performed independently six times) of histone extractions from 3T3-L1 cells transduced to express the indicated proteins. Membranes were probed with IDH1 and IDH2. (D) This is the same experiment as in Figure 7C. Representative TIC from samples harvested 7 days after transduction to express the indicated proteins in 3T-L1 cells. The detection of 2HG and glutamate are shown and based on spectra of derivatized commercial standards. Additional details can be found at https://osf.io/vfsbo/.

Figure 7—figure supplement 2.. Meta-analyses of effects from replication attempt of Lu et al., 2012.

Meta-analysis of each effect reported in Figure 7. Effect size and 95 % confidence intervals are presented for Lu et al., 2012, this replication study (Reproducibility Project: Cancer Biology, RP:CB), and a random-effects meta-analysis of those two effects. The effect size r is a standardized measure of the correlation (strength and direction) of the association between two variables and Cohen’s d is a standardized difference between the two indicated measurements where a larger value indicates a difference in expression between the two conditions. (A) Meta-analysis p values: H3K4me3 expression (p = 0.235), H3K9me2 expression (p = 0.117), H3K9me3 expression (p = 0.135), H3K36me3 expression (p = 0.133), and H3K79me2 expression (p = 0.090). (B) Meta-analysis p values: H3K9me2 expression between IDH1 WT and IDH1 R132H (p = 0.398), and H3K9me2 expression between IDH2 WT and IDH2 R172K (p = 0.202). (C) Meta-analysis p values: correlation of H3K4me3 expression and 2HG (p = 0.122), correlation of H3K9me2 expression and 2HG (p = 0.092), correlation of H3K9me3 expression and 2HG (p = 0.081), correlation of H3K36me3 expression and 2HG (p = 0.181), and correlation of H3K79me2 expression and 2HG (p = 0.185). Sample sizes used in Lu et al., 2012 and RP:CB are reported under the study name. Additional details can be found at https://osf.io/vfsbo/.