Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer

Abstract

Background: Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods.

Methods: We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years.

Results: In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09).

Conclusion: These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.

Figure 1:

Study flow diagram for HORIZON III participants and SEER-Medicare patients with first metastatic colorectal cancer. Superscripts indicate restrictions applied for A) consistency of treatment across populations, B) access to data and variables for use in the sampling model, and C) external positivity of the populations with respect to the primary risk factor of age.

Figure 2:

Kernel density plots for the predicted probability of sampling for bevacizumab and oxaliplatin-treated participants in the HORIZON trial (solid line) and the bevacizumab and oxaliplatin-treated 2004-2011 SEER-Medicare metastatic colorectal cancer target population (dashed line). The peaks are reflective of the categorical coding of several variables. Every joint combination of covariates contained at least one trial individual.

Figure 3:

Hazard ratios (HR) and survival curves comparing outcomes in the bevacizumab and oxaliplatin-treated 2004-2011 SEER-Medicare population and bevacizumab and oxaliplatin-treated HORIZON III patients before and after applying inverse odds of sampling weights. Weights were derived using sampling models containing age modeled with quadratic restricted splines with knots at the 20th, 40th, 60th, and 80th percentiles, sex, tumor differentiation (poor vs. moderate or well differentiated), and type of cancer. Panel A) shows the survival curves when targeting the full SEER-Medicare cohort. Panel B) shows the survival curves when targeting only individuals with predicted probability of frailty < 7.5%.