Assessing the Sensitivity of Treatment Effect Estimates to Differential Follow-Up Rates: Implications for Translational Research

Abstract

We develop a new tool for assessing the sensitivity of findings on treatment effectiveness to differential follow-up rates in the two treatment conditions being compared. The method censors the group with the higher response rate to create a synthetic respondent group that is then compared with the observed cases in the other condition to estimate a treatment effect. Censoring is done under various assumptions about the strength of the relationship between follow-up and outcomes to determine how informative differential dropout can alter inferences relative to estimates from models that assume the data are missing at random. The method provides an intuitive measure for understanding the strength of the association between outcomes and dropout that would be required to alter inferences about treatment effects. Our approach is motivated by translational research in which treatments found to be effective under experimental conditions are tested in standard treatment conditions. In such applications, follow-up rates in the experimental setting are likely to be substantially higher than in the standard setting, especially when observational data are used in the evaluation. We test the method on a case study evaluation of the effectiveness of an evidence-supported adolescent substance abuse treatment program (Motivational Enhancement Therapy/Cognitive Behavioral Therapy-5 [MET/CBT-5]) delivered by community-based treatment providers relative to its performance in a controlled research trial. In this case study, follow-up rates in the community based settings were extremely low (54%) compared to the experimental setting (95%) giving raise to concerns about non-ignorable drop-out.

Figure 1

Standardized mean differences (plus 95% confidence intervals) in outcomes among youth in the experimental setting (CYT) and propensity score weighted youth in the community-based sample (EAT).

Figure 2

Sensitivity analysis showing (i) the value of SSI for which the mean treatment effect estimate for the FF group found by comparing MET/CBT-5 among the community responders and synthetic experimental FF group across the 100 simulations was zero (i.e., gave treatment effect estimates indicating experimental and community settings were the same) and (ii) the value of SSI for which 95% of the simulations favored youth in the experimental setting (i.e., gave treatment effect estimates in favor of the experimental setting for the 2.5th and 97.5th percentiles). Note: SPS = Substance Problem Scale; SFS = Substance Frequency Scale; EPS = Emotional Problems Scale; IAS = Illegal Activities Scale