Assessing health care interventions via an interrupted time series model: Study power and design considerations

Abstract

The delivery and assessment of quality health care is complex with many interacting and interdependent components. In terms of research design and statistical analysis, this complexity and interdependency makes it difficult to assess the true impact of interventions designed to improve patient health care outcomes. Interrupted time series (ITS) is a quasi-experimental design developed for inferring the effectiveness of a health policy intervention while accounting for temporal dependence within a single system or unit. Current standardized ITS methods do not simultaneously analyze data for several units nor are there methods to test for the existence of a change point and to assess statistical power for study planning purposes in this context. To address this limitation, we propose the "Robust Multiple ITS" (R-MITS) model, appropriate for multiunit ITS data, that allows for inference regarding the estimation of a global change point across units in the presence of a potentially lagged (or anticipatory) treatment effect. Under the R-MITS model, one can formally test for the existence of a change point and estimate the time delay between the formal intervention implementation and the over-all-unit intervention effect. We conducted empirical simulation studies to assess the type one error rate of the testing procedure, power for detecting specified change-point alternatives, and accuracy of the proposed estimating methodology. R-MITS is illustrated by analyzing patient satisfaction data from a hospital that implemented and evaluated a new care delivery model in multiple units.

Keywords: change-point detection; complex interventions; patient satisfaction; power analysis; segmented regression; time series.

FIGURE 1

Plots the time series of observed average patient satisfaction for the Stroke and Surgical units. CNL, Clinical Nurse Leader

FIGURE 2

An example of a segmented regression model fit for the Stroke unit. The plot depicts (1) the segmented regression lines fit to the pre- and post–change-point phases, (2) the projection of the mean at the change point based on the pre–change-point regression, and (3) the change in level as defined here. The plot contains data from January 2010 to September 2010, instead of the entire observational period, to clearly illustrate the level change

FIGURE 3

Empirical power, over 10 000 iterations, for various number of units and for 4 regimes. The empirical power increases as the number of units and the length of time series increases, and the power increases as the adjacent correlation decreases

FIGURE 4

The proportion of estimated change points exactly equal to the true change point, over 10 000 iterations, for various number of units and for 4 regimes. Similar to the empirical power, the proportion of correctly estimated change points increases as the number of units and the length of time series increases

FIGURE 5

Plots the time series of observed average patient satisfaction for all hospital units, along with the estimated change point, estimated mean functions, and formal intervention time

FIGURE 6

Plots the time series of observed average patient satisfaction, along with the estimated change point, estimated mean functions, and formal intervention time for the Medical Surgery and cardiac units with and without observation t = 25 (January 2010), obtained by using Robust-ITS to conduct the unit-specific analyses. Note that the analysis with t = 25 is on the left and the analysis without t = 25 is on the right. CNL, Clinical Nurse Leader