An Approach to Assess Generalizability in Comparative Effectiveness Research: A Case Study of the Whole Systems Demonstrator Cluster Randomized Trial Comparing Telehealth with Usual Care for Patients with Chronic Health Conditions

Abstract

Background: Policy makers require estimates of comparative effectiveness that apply to the population of interest, but there has been little research on quantitative approaches to assess and extend the generalizability of randomized controlled trial (RCT)-based evaluations. We illustrate an approach using observational data.

Methods: Our example is the Whole Systems Demonstrator (WSD) trial, in which 3230 adults with chronic conditions were assigned to receive telehealth or usual care. First, we used novel placebo tests to assess whether outcomes were similar between the RCT control group and a matched subset of nonparticipants who received usual care. We matched on 65 baseline variables obtained from the electronic medical record. Second, we conducted sensitivity analysis to consider whether the estimates of treatment effectiveness were robust to alternative assumptions about whether "usual care" is defined by the RCT control group or nonparticipants. Thus, we provided alternative estimates of comparative effectiveness by contrasting the outcomes of the RCT telehealth group and matched nonparticipants.

Results: For some endpoints, such as the number of outpatient attendances, the placebo tests passed, and the effectiveness estimates were robust to the choice of comparison group. However, for other endpoints, such as emergency admissions, the placebo tests failed and the estimates of treatment effect differed markedly according to whether telehealth patients were compared with RCT controls or matched nonparticipants.

Conclusions: The proposed placebo tests indicate those cases when estimates from RCTs do not generalize to routine clinical practice and motivate complementary estimates of comparative effectiveness that use observational data. Future RCTs are recommended to incorporate these placebo tests and the accompanying sensitivity analyses to enhance their relevance to policy making.

Keywords: causal inference; chronic health conditions; external validity; generalizability; randomized trials; telehealth.

Figure 1

Patterns of emergency hospital admissions in the Whole Systems Demonstrator (WSD) telehealth trial (n = 3154). Figure reprinted with permission from Steventon A, Bardsley M, Billings J, et al. Effect of telehealth on use of secondary care and mortality: findings from the WSD cluster randomised trial. BMJ. 2012;344:e3874. http://www.bmj.com/content/344/bmj.e3874. Copyright © 2012, British Medical Journal Publishing Group.

Figure 2

Flow diagram showing practice and patient recruitment. We excluded (from both participant and nonparticipant groups) people without a continuous record of registration with one or more primary care practices over the 2 years preceding the trial period. As per the original study, we excluded trial participants who were recruited after September 2009 or not linked to routine data. In the current study, we also excluded the small number of participants who did not receive their allocated treatments.

Figure 3

Crude trends in rates of service use (contacts per patient per quarter). The observations to the left of the vertical line show rates of primary and secondary care contact for each of the 8 calendar quarters preceding trial recruitment (i.e., for a total of 2 years). The observations to the right of the vertical line show rates for survivors for the 4 quarters in the trial period. A gap has been imposed at the time of recruitment for clarity. Rates for RCT control patients (n = 1293) are shown in black. The dashed red line shows rates for the eligible nonparticipants (n = 88,830), while the solid red line shows rates for the matched subgroup of eligible nonparticipants (n = 1293). These were matched to RCT controls on variables including prior rates of primary and secondary care contact, and the placebo tests assess whether rates continued to be similar during the trial period. For the purposes of producing this figure, comparison patients were randomly allocated index dates in approximately the same distribution as trial control patients.