Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Feb 22:14:757-770.
doi: 10.2147/RMHP.S275831. eCollection 2021.

Three Statistical Approaches for Assessment of Intervention Effects: A Primer for Practitioners

Lihua Li  1   2   3 , Meaghan S Cuerden  4 , Bian Liu  1   3   5 , Salimah Shariff  6 , Arsh K Jain  4   6 , Madhu Mazumdar  1   2   3
Affiliations

Three Statistical Approaches for Assessment of Intervention Effects: A Primer for Practitioners

Lihua Li et al. Risk Manag Healthc Policy. .

Abstract

Introduction: Statistical methods to assess the impact of an intervention are increasingly used in clinical research settings. However, a comprehensive review of the methods geared toward practitioners is not yet available.

Methods and materials: We provide a comprehensive review of three methods to assess the impact of an intervention: difference-in-differences (DID), segmented regression of interrupted time series (ITS), and interventional autoregressive integrated moving average (ARIMA). We also compare the methods, and provide illustration of their use through three important healthcare-related applications.

Results: In the first example, the DID estimate of the difference in health insurance coverage rates between expanded states and unexpanded states in the post-Medicaid expansion period compared to the pre-expansion period was 5.93 (95% CI, 3.99 to 7.89) percentage points. In the second example, a comparative segmented regression of ITS analysis showed that the mean imaging order appropriateness score in the emergency department at a tertiary care hospital exceeded that of the inpatient setting with a level change difference of 0.63 (95% CI, 0.53 to 0.73) and a trend change difference of 0.02 (95% CI, 0.01 to 0.03) after the introduction of a clinical decision support tool. In the third example, the results from an interventional ARIMA analysis show that numbers of creatinine clearance tests decreased significantly within months of the start of eGFR reporting, with a magnitude of drop equal to -0.93 (95% CI, -1.22 to -0.64) tests per 100,000 adults and a rate of drop equal to 0.97 (95% CI, 0.95 to 0.99) tests per 100,000 per adults per month.

Discussion: When choosing the appropriate method to model the intervention effect, it is necessary to consider the structure of the data, the study design, availability of an appropriate comparison group, sample size requirements, whether other interventions occur during the study window, and patterns in the data.

Keywords: autoregressive integrated moving average; difference-in-difference; interrupted time series; segmented regression.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no conflict of interests for this work.

Figures

Figure 1
Figure 1
Trend in health insurance coverage in a low-income population in the United States from 2012 to 2017.
Figure 2
Figure 2
Mean scores and predicted regression line for appropriateness scores of imaging orders in an emergency department setting versus an inpatient setting over time.
Figure 3
Figure 3
Monthly numbers of 24-hour creatinine clearance collections in Ontario, Canada (adjusted for age and sex) and post-intervention forecast value.
See this image and copyright information in PMC

References

    1. Nayor M, Vasan RS. Recent update to the US cholesterol treatment guidelines. Circulation. 2016;133(18):1795–1806. doi:10.1161/CIRCULATIONAHA.116.021407 - DOI - PMC - PubMed
    1. McFarland DC, Shen M, Harris K, et al. ReCAP: would women with breast cancer prefer to receive an antidepressant for anxiety or depression from their oncologist? J Oncol Pract. 2016;12(2):172–174. doi:10.1200/JOP.2015.006833 - DOI - PMC - PubMed
    1. Garg P, Ko DT, Jenkyn KMB, Li L, Shariff SZ. Infective endocarditis hospitalizations and antibiotic prophylaxis rates before and after the 2007 American Heart Association guideline revision. Circulation. 2019;140(3):170–180. doi:10.1161/CIRCULATIONAHA.118.037657 - DOI - PubMed
    1. Han B, Yu H. Causal difference-in-differences estimation for evaluating the impact of semi-continuous medical home scores on health care for children. Health Serv Outcomes Res Methodol. 2019;19(1):61–78. doi:10.1007/s10742-018-00195-9 - DOI - PMC - PubMed
    1. Kagoma YK, Garg AX, Li L, Jain AK. Reporting of the estimated glomerular filtration rate decreased creatinine clearance testing. Kidney Int. 2012;81(12):1245–1247. doi:10.1038/ki.2011.483 - DOI - PubMed