Development of a practical approach to expert elicitation for randomised controlled trials with missing health outcomes: Application to the IMPROVE trial

Abstract

Background/aims: The analyses of randomised controlled trials with missing data typically assume that, after conditioning on the observed data, the probability of missing data does not depend on the patient's outcome, and so the data are 'missing at random' . This assumption is usually implausible, for example, because patients in relatively poor health may be more likely to drop out. Methodological guidelines recommend that trials require sensitivity analysis, which is best informed by elicited expert opinion, to assess whether conclusions are robust to alternative assumptions about the missing data. A major barrier to implementing these methods in practice is the lack of relevant practical tools for eliciting expert opinion. We develop a new practical tool for eliciting expert opinion and demonstrate its use for randomised controlled trials with missing data.

Methods: We develop and illustrate our approach for eliciting expert opinion with the IMPROVE trial (ISRCTN 48334791), an ongoing multi-centre randomised controlled trial which compares an emergency endovascular strategy versus open repair for patients with ruptured abdominal aortic aneurysm. In the IMPROVE trial at 3 months post-randomisation, 21% of surviving patients did not complete health-related quality of life questionnaires (assessed by EQ-5D-3L). We address this problem by developing a web-based tool that provides a practical approach for eliciting expert opinion about quality of life differences between patients with missing versus complete data. We show how this expert opinion can define informative priors within a fully Bayesian framework to perform sensitivity analyses that allow the missing data to depend upon unobserved patient characteristics.

Results: A total of 26 experts, of 46 asked to participate, completed the elicitation exercise. The elicited quality of life scores were lower on average for the patients with missing versus complete data, but there was considerable uncertainty in these elicited values. The missing at random analysis found that patients randomised to the emergency endovascular strategy versus open repair had higher average (95% credible interval) quality of life scores of 0.062 (-0.005 to 0.130). Our sensitivity analysis that used the elicited expert information as pooled priors found that the gain in average quality of life for the emergency endovascular strategy versus open repair was 0.076 (-0.054 to 0.198).

Conclusion: We provide and exemplify a practical tool for eliciting the expert opinion required by recommended approaches to the sensitivity analyses of randomised controlled trials. We show how this approach allows the trial analysis to fully recognise the uncertainty that arises from making alternative, plausible assumptions about the reasons for missing data. This tool can be widely used in the design, analysis and interpretation of future trials, and to facilitate this, materials are available for download.

Keywords: Bayesian analysis; Missing data; clinical trials; expert elicitation; pattern-mixture models; quality of life; sensitivity analysis.

Figure 1.

Illustration of the estimation of treatment effectiveness using a pattern-mixture model that allows for outcome data to be MNAR. $\mu$ represents the mean QoL for patients who returned their QoL questionnaires, $\delta$ represents the difference in the mean QoL between patients who did and did not return their QoL questionnaires and $\pi$ represents the proportion of patients who did not return their QoL questionnaires. E and O indicate the eEVAR and open repair treatment groups respectively. Simple arithmetic example that uses hypothetical elicited values to re-calculate the effectiveness of eEVAR versus open repair on QoL score. The example uses a pattern-mixture model to allow for data that are MNAR. Information from QoL data that are observed in the RCT: sample mean (SE) QoL score for patients who completed QoL questionnaire $\mathrm{eEVAR\; strategy}=0.76(0.02),\mathrm{open\; repair\; strategy}=0.69(0.03)$ proportion of patients who did not return their QoL questionnaire $\mathrm{eEVAR\; strategy}=0.18,\mathrm{open\; repair\; strategy}=0.24$ Information elicited from an expert: mean (SD) of difference in mean QoL between patients who did and did not return their QoL questionnaire $\mathrm{eEVAR\; strategy}=-0.01(0.04),\mathrm{open\; repair\; strategy}=-0.05(0.1)$ Then a point estimate of the treatment difference can be calculated as $({\mu }_E+{\pi }_E{\delta }_E)-({\mu }_O+{\pi }_O{\delta }_O)=(0.76-0.18\times 0.01)-(0.69-0.24\times 0.05)=0.08.$ Assuming independence between variables, the variance (V) of the treatment difference is $V({\mu }_E)+{\pi }_E^{2}V({\delta }_E)+V({\mu }_O)+{\pi }_O^{2}V({\delta }_O)=0.{02}^2+0.{18}^{2}0.{04}^2+0.{03}^2+0.{24}^{2}0.1^2=0.002,$ and a 95% confidence interval (CI) for the treatment difference can be estimated as $(0.08-1.96\times 0.044,0.08+1.96\times 0.044)=(-0.01,0.17)$ Hence using a pattern-mixture model with expert information reports an estimate of the effectiveness of eEVAR versus open repair of treatment difference (95% CI) of 0.08 (−0.01, 0.17), compared to 0.07 (0.00, 0.14) for calculations based on the observed QoL alone. Note the wider confidence interval from using the pattern-mixture model, as this approach takes account of the uncertainty from the missing data that may be MNAR.

Figure 2.

Screen shots from the elicitation tool.

Figure 3.

Individual and pooled prior distributions for patients randomised to eEVAR and open repair arms: (a) eEVAR: all experts and (b) OPEN: all experts. Thin grey lines = individual priors, thick black lines = smoothed pooled priors across all experts. Although each individual prior has been elicited as a normal distribution, this restriction does not apply to the pooled priors which are a mixture of normal distributions.

Figure 4.

Difference in mean quality of life score at 3 months between randomised arm (eEVAR - open repair) for survivors. Each shaded rectangular strip shows the full posterior distribution of the difference in mean QoL at 3 months for survivors for one model run. The darkness at a point is proportional to the probability density, such that the strip is darkest at the maximum density and fades into the background at the minimum density. The posterior mean and 95% credible interval are marked. *the posterior probability that the eEVAR QoL at 3 months is at least 0.03 greater than the open repair QoL. 0.03 is the minimum clinically important difference.