Bias modelling in evidence synthesis

Abstract

Policy decisions often require synthesis of evidence from multiple sources, and the source studies typically vary in rigour and in relevance to the target question. We present simple methods of allowing for differences in rigour (or lack of internal bias) and relevance (or lack of external bias) in evidence synthesis. The methods are developed in the context of reanalysing a UK National Institute for Clinical Excellence technology appraisal in antenatal care, which includes eight comparative studies. Many were historically controlled, only one was a randomized trial and doses, populations and outcomes varied between studies and differed from the target UK setting. Using elicited opinion, we construct prior distributions to represent the biases in each study and perform a bias-adjusted meta-analysis. Adjustment had the effect of shifting the combined estimate away from the null by approximately 10%, and the variance of the combined estimate was almost tripled. Our generic bias modelling approach allows decisions to be based on all available evidence, with less rigorous or less relevant studies downweighted by using computationally simple methods.

Fig. 1

Identifying internal and external biases

Fig. 2

Elicitation scale for quantifying additive bias

Fig. 3

Effect of ranges for bias on the approximate width of the CI for the bias-adjusted log-odds-ratio, assuming rare events and no intervention effect (ranges refer to a symmetric relative risk scale, as shown in Fig. 2): , no bias; , 67% range (0.9, 0.9); , 67% range (0.7, 0.7); 67% range (0.5, 0.5)

Fig. 4

Additive biases in Hermann et al. (1984), 67% ranges of distributions elicited from four assessors (A–D) and means and 67% ranges for total additive bias

Fig. 5

Effect of adjusting for (a) additive bias and (b) all bias on the odds ratio (with 95% CIs) in Hermann et al. (1984)

Fig. 6

Meta-analysis of eight studies evaluating the effectiveness of routine anti-D prophylaxis—unadjusted and bias-adjusted odds ratios (with 95% CIs) (for each result, the corresponding total ‘effective number of events’ is listed alongside): (a) unadjusted; (b) bias adjusted (additive); (c) bias adjusted (all)

Fig. 7

Elicitation scale for quantifying proportional bias

Fig. 8

Proportional biases in Hermann et al. (1984), 67% ranges of distributions elicited from four assessors (A–D) and means and 67% ranges for total internal and external proportional bias

Fig. 9

Meta-analysis of anti-D immunoglobulin studies adjusted for additive and proportional biases, using the elicited opinions of assessors (a) A, (b) B, (c) C and (d) D separately (with 95% CIs)