Double Sampling for Informatively Missing Data in Electronic Health Record-Based Comparative Effectiveness Research

Abstract

Missing data arise in most applied settings and are ubiquitous in electronic health records (EHR). When data are missing not at random (MNAR) with respect to measured covariates, sensitivity analyses are often considered. These solutions, however, are often unsatisfying in that they are not guaranteed to yield actionable conclusions. Motivated by an EHR-based study of long-term outcomes following bariatric surgery, we consider the use of double sampling as a means to mitigate MNAR outcome data when the statistical goals are estimation and inference regarding causal effects. We describe assumptions that are sufficient for the identification of the joint distribution of confounders, treatment, and outcome under this design. Additionally, we derive efficient and robust estimators of the average causal treatment effect under a nonparametric model and under a model assuming outcomes were, in fact, initially missing at random (MAR). We compare these in simulations to an approach that adaptively estimates based on evidence of violation of the MAR assumption. Finally, we also show that the proposed double sampling design can be extended to handle arbitrary coarsening mechanisms, and derive nonparametric efficient estimators of any smooth full data functional.

Keywords: causal inference; double sampling; missing data; semiparametric theory; study design.

FIGURE 1

Schematic illustrating the double sampling study design. Gray boxes indicate subgroups for which the outcome is observed: either subjects are observed in the EHR directly ($R=1$) or are selected in the follow‐up sample ($R=0$ and $S=1$).

FIGURE 2

Simulation results for experiments of Section 5.1, with (a) no violation of MAR (${\beta }_{RA}=0$); (b) a moderate violation of MAR (${\beta }_{RA}=0.016$); and (c) a large violation of MAR (${\beta }_{RA}=0.032$). S#1 = Strategy #1, augmented IPW‐based estimator assuming MAR, ${\widehat{\xi }}_1-{\widehat{\xi }}_0$; S#1‐OR = outcome regression‐based estimator assuming MAR; S#1‐IPW = IPW‐based estimator assuming MAR; S#2 = Strategy #2, influence‐function based estimator using double sampling, ${\widehat{\tau }}_1-{\widehat{\tau }}_0$; S#2‐PW = S#2 but with ${\widehat{\pi }}_a$ misspecified; S#2‐MW = S#2‐DS but with $({\widehat{\mu }}_{a,S},{\widehat{\mu }}_{a,R},{\widehat{\gamma }}_a)$ misspecified; S#2‐BW = S#2 but with both $({\widehat{\mu }}_{a,S},{\widehat{\mu }}_{a,R},{\widehat{\gamma }}_a)$ and ${\widehat{\pi }}_a$ misspecified; S#3 = Strategy #3, semiparametric efficient estimator under MAR, ${\widehat{\tau }}_1^{\ast }-{\widehat{\tau }}_0^{\ast }$. ATE = average treatment effect; the red dashed line indicates the true ATE.

FIGURE 3

Simulation results for experiments of Section 5.2, comparing Strategy #2 of the nonparametric influence function‐based estimator (${\widehat{\tau }}_1-{\widehat{\tau }}_0$), Strategy #3 of the semiparametric efficient estimator under MAR (${\widehat{\tau }}_1^{\ast }-{\widehat{\tau }}_0^{\ast }$), Strategy #4 of the ad hoc approach described in Section 5.1, and Strategy #5 of the adaptive estimator of Rothenhäusler [32] (${\widehat{\tau }}_1^{†}-{\widehat{\tau }}_0^{†}$). Subplot (a) shows the empirical mean squared error (MSE) of each approach, divided by the MSE of the nonparametric efficient estimator. Subplot (b) shows the empirical variance of each approach, divided by the variance of the nonparametric efficient estimator. Subplot (c) shows the empirical bias of each approach.

FIGURE 4

Results for data application in Section 6, showing the estimated effect on percent total weight change at 3 years. Subplots (a), (b), and (c) correspond to follow‐up subsamples of size 500, 1000, and 1500 individuals. Blue and red lines refer to estimated 95% confidence intervals when outcomes are initially MAR, and MNAR, respectively. Point estimates are marked with black dots. Strategy #1 = augmented‐IPW estimator with incomplete data only, assuming MAR; Strategy #2 = influence‐function based estimator using double sampling, ${\widehat{\tau }}_1-{\widehat{\tau }}_0$; Strategy #3 = semiparametric efficient estimator under MAR, ${\widehat{\tau }}_1^{\ast }-{\widehat{\tau }}_0^{\ast }$; Strategy #5 = adaptive estimator of Rothenhäusler [32], ${\widehat{\tau }}_1^{†}-{\widehat{\tau }}_0^{†}$. The black dashed line represents the point estimate for the benchmark full data analysis.