DICE 2: a further investigation of the effects of chance in life, death and subgroup analyses

Abstract

In an investigation into how chance might influence the distribution of deaths in a randomised trial and the time of those deaths, and to highlight the possible dangers of subgroup analyses, 100 randomised controlled trials were simulated and 50 subgroup pairs were simulated for some of these trials. Each of 580 control patients from a colorectal cancer trial was randomly coded to simulate allocation to treatment or control, the main outcome measure being time to death. Not surprisingly, most of the 100 trials gave non-significant results. Four were conventionally significant with logrank 2p-values of less than 0.05. The most extreme result was associated with a logrank 2p-value of 0.003, showing an absolute reduction in four-year mortality of 40% (SD 15) for patients allocated to treatment. One of the simulated prognostic factors for this trial (subgroup 13) showed that mortality for one type of patient was non-significantly slightly increased by treatment, whereas treatment reduced four-year mortality by 64% (SD 16) among the other patients in the trial (2p = 0.00006). Similar, extreme results were found for a trial of borderline statistical significance overall. Chance can influence the overall results of any randomised controlled trial, regardless of how well it is conducted, and can play an even more powerful role in the results of subgroup analyses. This should be borne in mind both by trialists when reporting their results and by readers and reviewers of those reports.