Comparison of anticipated and actual control group outcomes in randomised trials in paediatric oncology provides evidence that historically controlled studies are biased in favour of the novel treatment

Abstract

Background: Historically controlled studies are commonly undertaken in paediatric oncology, despite their potential biases. Our aim was to compare the outcome of the control group in randomised controlled trials (RCTs) in paediatric oncology with those anticipated in the sample size calculations in the protocols. Our rationale was that, had these RCTs been performed as historical control studies instead, the available outcome data used to calculate the sample size in the RCT would have been used as the historical control outcome data.

Methods: A systematic search was undertaken for published paediatric oncology RCTs using the Cochrane Central Register of Controlled Trials (CENTRAL) database from its inception up to July 2013. Data on sample size assumptions and observed outcomes (timetoevent and proportions) were extracted to calculate differences between randomised and historical control outcomes, and a one-sample t-test was employed to assess whether the difference between anticipated and observed control groups differed from zero.

Results: Forty-eight randomised questions were included. The median year of publication was 2005, and the range was from 1976 to 2010. There were 31 superiority and 11 equivalence/noninferiority randomised questions with time-to-event outcomes. The median absolute difference between observed and anticipated control outcomes was 5.0% (range: -23 to +34), and the mean difference was 3.8% (95% CI: +0.57 to +7.0; P = 0.022).

Conclusions: Because the observed control group (that is, standard treatment arm) in RCTs performed better than anticipated, we found that historically controlled studies that used similar assumptions for the standard treatment were likely to overestimate the benefit of new treatments, potentially leading to children with cancer being given ineffective therapy that may have additional toxicity.

Figure 1

Flowchart of the trial selection process. RCT, Randomised controlled trial; RQ, Randomised question. Note: () represent publications and [] represent RQs.

Figure 2

Distribution of absolute differences between observed and anticipated control (%). RQ, Randomised question.

Figure 3

Scatter plot for control difference and disease prognosis (100% implies a good prognosis).

Figure 4

Absolute differences for randomised question (RQ; left) and historical control (right) comparison.

Figure 5

Relative differences for historical control comparison and randomised comparison. Solid dots, triangles or squares represent the hazard ratios (HRs) or risk ratios (RRs) for a randomised comparison. The crosses represent the HRs or RRs for the historical control comparison (HCC). The solid vertical lines show the median HRs or RRs for the randomised comparison. The dashed vertical lines show the median HRs or RRs for the HCC. For example, for the 48th question, the HR for the randomised comparison is 1.66, whereas it is 2.29 for the HCC.