Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach?

Abstract

Identifying which individuals benefit most from particular treatments or other interventions underpins so-called personalised or stratified medicine. However, single trials are typically underpowered for exploring whether participant characteristics, such as age or disease severity, determine an individual’s response to treatment. A meta-analysis of multiple trials, particularly one where individual participant data (IPD) are available, provides greater power to investigate interactions between participant characteristics (covariates) and treatment effects. We use a published IPD meta-analysis to illustrate three broad approaches used for testing such interactions. Based on another systematic review of recently published IPD meta-analyses, we also show that all three approaches can be applied to aggregate data as well as IPD. We also summarise which methods of analysing and presenting interactions are in current use, and describe their advantages and disadvantages. We recommend that testing for interactions using within-trials information alone (the deft approach) becomes standard practice, alongside graphical presentation that directly visualises this.

Fig 1 Use of a deluded approach to analyse and present interactions in meta-analysis , illustrating how the effect of an early supported hospital discharge (ESD) strategy might vary by whether a carer is present. Left panel presents the effect of ESD for each subgroup within each trial, but ordered by subgroup; and right panel presents just the meta-analysed effects for each subgroup. The difference between the effects in right panel gives a deluded analysis (mean difference of 2.23, 95% confidence interval −2.82 to 7.28, P=0.39). Sizing of squares are in proportion to the inverse of the variance of the estimates. Note that the subgroup meta-analysis estimates do not match exactly those originally reported, because we used a fixed effect model for simplicity, rather than the random-effects model used by the original authors. See also web appendix A3 for references of studies and appendix B2 for details of statistical reanalysis

Fig 2 Use of a deft approach to analyse and present interactions in meta-analysis , illustrating how the effect of an early supported hospital discharge (ESD) strategy might vary by whether a carer is present. The left panel again presents the effect of ESD for each subgroup within each trial, but now ordered by trial. The right panel shows the interactions between the effect of ESD and presence of a carer for each trial, along with a meta-analysis of the interaction estimates (mean difference –6.64, 95% confidence interval –13.65 to 0.71, P=0.77; heterogeneity of interaction estimates: Q=12.8, df=7, I²=45%). Daft and deluded interaction estimates are presented alongside for comparison. Squares are used to depict treatment effect and circles the interaction effects, with sizing in proportion to the inverse of the variance of the estimates

Fig 3 Scatter plot (logit scale) of P values from 31 deft reanalyses (see web appendix B) of treatment-covariate interactions against the corresponding daft reanalyses. Added lines are at P=0.1; arguably a suitable level of significance for an interaction test for which a trial was not powered

Fig 4 Bland-Altman plot showing level of agreement between treatment-covariate interactions from deluded and deft analyses. Thirty one interactions were reanalysed, but only 26 with outcomes measured by hazard ratios or odds ratios were plotted. The remaining five interactions, including our illustrative example, could not be included since their outcomes were measured by mean differences and were hence incompatible. Treatment-covariate interactions (measured on the log scale) might have a positive or a negative sign, but in this plot they have all been set to negative. Hence, differences in interaction effects below the zero line represent cases where a deluded analysis gives a result in the same direction as, but more extreme than, the equivalent deft analysis, and vice versa. Shaded area=Bland-Altman 95% limits of agreement¹⁴; solid line represents mean difference (bias); dashed lines are 95% confidence intervals around the mean difference