Reweighting Randomized Controlled Trial Evidence to Better Reflect Real Life - A Case Study of the Innovative Medicines Initiative

Abstract

Evidence from randomized controlled trials available for timely health technology assessments of new pharmacological treatments and regulatory decision making may not be generalizable to local patient populations, often resulting in decisions being made under uncertainty. In recent years, several reweighting approaches have been explored to address this important question of generalizability to a target population. We present a case study of the Innovative Medicines Initiative to illustrate the inverse propensity score reweighting methodology, which may allow us to estimate the expected treatment benefit if a clinical trial had been run in a broader real-world target population. We learned that identifying treatment effect modifiers, understanding and managing differences between patient characteristic data sets, and balancing the closeness of trial and target patient populations with effective sample size are key to successfully using this methodology and potentially mitigating some of this uncertainty around local decision making.

Figure 1

Flowcharts for JMDB randomized controlled trial and FRAME observational study patient populations following application of eligibility criteria for this case study illustration. ECOG, Eastern Cooperative Oncology Group.

Figure 2

Distribution of propensity scores for patients in JMDB randomized controlled trial and FRAME observational study (unstandardized untrimmed primary analysis).

Figure 3

Standardized difference plot for original unweighted and inverse propensity score‐weighted differences between studies (JMDB randomized controlled trial vs. FRAME observational study). Standardized difference plot ordered by magnitude of difference between JMDB randomized controlled trial vs. FRAME observational study before and after inverse propensity score weighting. ECOG, Eastern Cooperative Oncology Group; NSCLC, non‐small cell lung cancer.

Figure 4

Survival curves for pemetrexed and gemcitabine arms of JMDB randomized controlled trial population included in this case study illustration: (a) original unweighted Kaplan–Meier curves; (b) original unweighted predicted Cox proportional hazards model survivor functions; and (c) corresponding inverse propensity score‐weighted predicted Cox proportional hazards model survivor functions (unstandardized untrimmed primary analysis). Panels b and c show survivor functions with 95% confidence limits.