Completeness of reporting and risks of overstating impact in cluster randomised trials: a systematic review

Abstract

Overstating the impact of interventions through incomplete or inaccurate reporting can lead to inappropriate scale-up of interventions with low impact. Accurate reporting of the impact of interventions is of great importance in global health research to protect scarce resources. In global health, the cluster randomised trial design is commonly used to evaluate complex, multicomponent interventions, and outcomes are often binary. Complete reporting of impact for binary outcomes means reporting both relative and absolute measures. We did a systematic review to assess reporting practices and potential to overstate impact in contemporary cluster randomised trials with binary primary outcome. We included all reports registered in the Cochrane Central Register of Controlled Trials of two-arm parallel cluster randomised trials with at least one binary primary outcome that were published in 2017. Of 73 cluster randomised trials, most (60 [82%]) showed incomplete reporting. Of 64 cluster randomised trials for which it was possible to evaluate, most (40 [63%]) reported results in such a way that impact could be overstated. Care is needed to report complete evidence of impact for the many interventions evaluated using the cluster randomised trial design worldwide.

Figure 1:. Summary of reporting of results for CRTs with primary binary outcome (for N=73 CRTs included in systematic review)

Abbreviation: NNT – Number Needed to Treat. Note: Percentages above each bar are out of 73, except for the final bar, which is out of 64. Percentages of segments within each bar sum to 100%, except for absolute measure type.

1. 1 reported proportions per arm with a p-value; 4 reported only proportions per arm with neither a p-value nor other measure of statistical significance; 1 reported only a p-value with no proportions by arm.

2. 2 articles reported “difference-in-differences” as the between-arm (i.e. intervention vs. control) difference in the within-arm change in proportion from baseline to endline.

3. 2 articles reported a ratio of odds ratios (ROR) in the abstract and main text. More specifically, 1 ROR was a comparison between intervention and control arms of the within-arm odds ratio for baseline to endline change, and 1 ROR was the ratio of the between-arm odds ratio (i.e. intervention effect) based on two levels of a postrandomization covariate.

4. Categories are not mutually exclusive. One paper reports both a NNT and a risk difference.

5. For 64 articles that report both an intervention effect as well as outcome proportions by arm. Note that, of the 73 articles, 6 report no intervention effect (neither absolute nor relative effect) and an additional 3 articles do not report outcome proportions by arm and therefore outcome magnitude cannot be ascertained.

6. Common outcome defined as: risk of the primary binary outcome is > 10% in both the intervention arm and the control arm.

7. Rare outcome defined as: risk of the primary binary outcome is ≤ 10% in either the intervention arm or the control arm.

8. CRTs are classified as having potential for overstatement of intervention impact if they either:

   1. Report only a relative measure (i.e. with no absolute measure) when the outcome is rare, or,

   2. Report an odds ratio when the outcome is common.

9. Among the 64 articles that report both an intervention effect as well as outcome proportions by arm.

Figure 2:. Magnitude of relative effect in relation to reference risk for CRTs with primary binary outcome (for N=59¹ CRTs in systematic review that reported a relative effect)

1. 55 of the 59 articles reporting relative measures are included in this figure. Four articles cannot be included for the following reasons:

   1. 1 article with rare outcome that reports only a relative measure has an odds ratio of 512.

   2. 1 article with rare outcome reports risk ratios, but only stratified by gender.

   3. 2 articles with common outcome each report a ratio of odds ratio (ROR): 1 ROR was a ratio for intervention arm vs. control arm of the within-arm odds ratio for baseline to endline change, and 1 ROR was the ratio between two levels of a post-randomization covariate of the between-arm odds ratio (intervention vs. control) of the intervention effect.

2. CRTs classified as having the potential for intervention impact to be overstated are shown using orange symbols. These are for articles that either:

   1. Report only a relative measure (i.e. with no absolute measure) when the outcome is rare (defined as risk ≤10% in either arm, or, equivalently, reference risk ≤10%), or,

   2. Report an odds ratio when the outcome is common (defined as risk >10% in both arms, equivalently reference risk >10%).

3. 28 studies reported an odds ratio and had a common outcome (risk > 10% in both arms). The magnitude of potential for overstatement of impact was quantified via the ratio of the odds ratio (OR) and the risk ratio (RR). This was estimated for each CRT using the reported risks by arm to obtain both the OR and RR, from which the ratio OR/RR was calculated. Mean (SD) [min, 25^th, 50^th, 75^th percentiles, max] ratio: 1.4 (0.6) [1, 1.06, 1.21, 1.43, 3.2].

4. For ease of visualization, the horizontal axis shows the reference risk, which is the smaller of the reported intervention-arm and control-arm risks.