A meta-analysis of threats to valid clinical inference in preclinical research of sunitinib

Abstract

Poor study methodology leads to biased measurement of treatment effects in preclinical research. We used available sunitinib preclinical studies to evaluate relationships between study design and experimental tumor volume effect sizes. We identified published animal efficacy experiments where sunitinib monotherapy was tested for effects on tumor volume. Effect sizes were extracted alongside experimental design elements addressing threats to valid clinical inference. Reported use of practices to address internal validity threats was limited, with no experiments using blinded outcome assessment. Most malignancies were tested in one model only, raising concerns about external validity. We calculate a 45% overestimate of effect size across all malignancies due to potential publication bias. Pooled effect sizes for specific malignancies did not show apparent relationships with effect sizes in clinical trials, and we were unable to detect dose-response relationships. Design and reporting standards represent an opportunity for improving clinical inference.

Keywords: cancer; epidemiology; global health; human biology; medicine; meta-analysis; mouse; preclinical; rat; systematic review.

Figure 1.. Descriptive analysis of (A) internal, construct, and (B) external validity design elements.

External validity scores were calculated for each malignancy type tested, according to the formula: number species used + number of models used; an extra point was assigned if a malignancy type tested more than one species and more than one model. DOI: http://dx.doi.org/10.7554/eLife.08351.003

Figure 1—figure supplement 1.. Descriptive analysis of (A) internal, construct, and (B) external validity design elements for all experiments (n = 332) extracted for validity data parameters.

DOI: http://dx.doi.org/10.7554/eLife.08351.005

Figure 2.. Summary of pooled SMDs for each malignancy type.

Shaded region denotes the pooled standardized mean difference (SMD) and 95% confidence interval (CI) (−1.8 [−2.1, −1.6]) for all experiments combined at the last common time point (LCT). DOI: http://dx.doi.org/10.7554/eLife.08351.008

Figure 2—figure supplement 1.. Effect sizes for all included experiments (n = 158).

DOI: http://dx.doi.org/10.7554/eLife.08351.010

Figure 3.. Relationship between study design elements and effect sizes.

The shaded region denotes the pooled SMD and 95% CI (−1.8 [−2.1, −1.6]) for all experiments combined at the LCT. DOI: http://dx.doi.org/10.7554/eLife.08351.011

Figure 4.. Funnel plot to detect publication bias.

Trim and fill analysis was performed on pooled malignancies, as well as the three malignancies with the greatest study volume. (A) All experiments for all malignancies (n = 182), (B) all experiments within renal cell carcinoma (RCC) (n = 35), (C) breast cancer (n = 32), and (D) colorectal cancer (n = 29). Time point was the LCT. Open circles denote original data points whereas black circles denote ‘filled’ experiments. Trim and fill did not produce an estimate in RCC; therefore, no overestimation of effect size could be found. DOI: http://dx.doi.org/10.7554/eLife.08351.012

Figure 5.. Dose–response curves for sunitinib preclinical studies.

Only experiments with a once daily (no breaks) administration schedule were included in both graphs. Effect size data were taken from a standardized time point (14 days after first sunitinib administration). (A) Experiments (n = 158) from all malignancies tested failed to show a dose–response relationship. (B) A dose–response relationship was not detected for RCC (n = 24). (C) Dose–response curves reported in individual studies within the RCC subset showed dose–response patterns (blue diamond = Huang 2010a [n = 3], red square = Huang 2010d [n = 3], green triangle = Ko 2010a [n = 3], purple X = Xin 2009 [n = 3]). DOI: http://dx.doi.org/10.7554/eLife.08351.013