Estimating relative risks and risk differences in randomised controlled trials: a systematic review of current practice

Abstract

Background: Guidelines for randomised controlled trials (RCTs) recommend reporting relative and absolute measures of effect for binary outcomes while adjusting for covariates. There are a number of different ways covariate-adjusted relative risks and risk differences can be estimated.

Objectives: Our goal was to identify methods used to estimate covariate-adjusted relative risk and risk differences in RCTs published in high-impact journals with binary outcomes. Other secondary objectives included the identification of how covariates are chosen for adjustment and whether covariate adjustment results in an increase in statistical precision in practice.

Methods: We included two-arm parallel RCTs published in JAMA, NEJM, Lancet, or the BMJ between January 1, 2018, and March 11, 2023, reporting relative risks or risk differences as a summary measure for a binary primary outcome. The search was conducted in Ovid-MEDLINE.

Results: Of the 308 RCTs identified, around half (49%; 95% CI: 43-54%) reported a covariate-adjusted relative risk or risk difference. Of these, 82 reported an adjusted relative risk. When the reporting was clear (n = 65, 79%), the log-binomial model (used in 65% of studies; 95% CI: 52-76%) and modified Poisson (29%; 95% CI: 19-42%) were most commonly used. Of the 92 studies that reported an adjusted risk difference, when the reporting was clear (n = 56, 61%), the binomial model (used in 48% of studies; 95% CI: 35-62%) and marginal standardisation (21%; 95% CI: 12-35%) were the common approaches used.

Conclusions: Approximately half of the RCTs report either a covariate-adjusted relative risk or risk difference. Many RCTs lack adequate details on the methods used to estimate covariate-adjusted effects. Of those that do report the approaches used, the binomial model, modified Poisson and to a lesser extent marginal standardisation are the approaches used.

Keywords: Binary outcomes; Covariate adjustment; Relative risk; Risk difference; Statistical efficiency; Statistical practice.

Fig. 1

PRISMA flow diagram of the selection process for eligible studies. The overall review sample includes RCTs published between January 1, 2018, and March 11, 2023, in selected high-impact journals, with a primary binary outcome, that report either a relative risk (RR) or risk difference (RD) (unadjusted or adjusted) as a summary measure. The numbers reported for unadjusted RR and RD are not mutually exclusive to the reporting of adjusted RR and RD. RR, relative risk; RD, risk difference. The nested review sample is a sub-sample of RCTs identified from the overall review sample that reports a covariate-adjusted RR or RD for the primary, secondary, or exploratory analysis. The numbers reported for adjusted RR and RD are not mutually exclusive

Fig. 2

Comparison of adjusted to unadjusted relative risks. Footnotes: relative risk; CIs, confidence intervals. If the point falls below the line of equality, then the value is smaller under the adjusted analysis than under the unadjusted analysis (as anticipated per theory). However, if the point is above the line of equality, then the value is larger under the adjusted analysis than under the unadjusted analysis

Fig. 3

Comparison of adjusted to unadjusted risk differences. Footnotes: RD, risk difference; CIs, confidence intervals. If the point falls below the line of equality, then the value is smaller under the adjusted analysis than under the unadjusted analysis (as anticipated per theory). However, if the point is above the line of equality, then the value is larger under the adjusted analysis than under the unadjusted analysis