Identifying appropriate comparison groups for health system interventions in the COVID-19 era

Abstract

Introduction: COVID-19 has created additional challenges for the analysis of non-randomized interventions in health system settings. Our objective is to evaluate these challenges and identify lessons learned from the analysis of a medically tailored meals (MTM) intervention at Kaiser Permanente Northwest (KPNW) that began in April 2020.

Methods: We identified both a historical and concurrent comparison group. The historical comparison group included patients living in the same area as the MTM recipients prior to COVID-19. The concurrent comparison group included patients admitted to contracted non-KPNW hospitals or admitted to a KPNW facility and living outside the service area for the intervention but otherwise eligible. We used two alternative propensity score methods in response to the loss of sample size with exact matching to evaluate the intervention.

Results: We identified 452 patients who received the intervention, 3873 patients in the historical comparison group, and 5333 in the concurrent comparison group. We were able to mostly achieve balance on observable characteristics for the intervention and the two comparison groups.

Conclusions: Lessons learned included: (a) The use of two different comparison groups helped to triangulate results; (b) the meaning of utilization measures changed pre- and post-COVID-19; and (c) that balance on observable characteristics can be achieved, especially when the comparison groups are meaningfully larger than the intervention group. These findings may inform the design for future evaluations of interventions during COVID-19.

Keywords: COVID‐19; comparison groups; observational studies; program evaluation.

FIGURE 1

(A): Common support for intervention and historical comparison. (B): Common support for intervention and concurrent comparison

FIGURE 2

(A): Standardized differences after matching for historical comparison group. (B): Standardized differences after matching for concurrent comparison group. CHF, stands for congestive heart failure; CKD, stands for chronic kidney disease; COPD, stands for Chronic obstructive pulmonary disease; ER, stands for emergency room

FIGURE 3

(A): Standardized differences after weighting for historical comparison group. (B): Standardized differences after weighting for concurrent comparison group. CHF, stands for congestive heart failure; CKD, stands for chronic kidney disease; COPD, stands for Chronic obstructive pulmonary disease; ER, stands for emergency room