Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2017 Feb;32(2):204-209.
doi: 10.1007/s11606-016-3841-9. Epub 2016 Oct 18.

False Dichotomies and Health Policy Research Designs: Randomized Trials Are Not Always the Answer

Stephen B Soumerai  1 , Rachel Ceccarelli  2 , Ross Koppel  3
Affiliations

False Dichotomies and Health Policy Research Designs: Randomized Trials Are Not Always the Answer

Stephen B Soumerai et al. J Gen Intern Med. 2017 Feb.

Abstract

Some medical scientists argue that only data from randomized controlled trials (RCTs) are trustworthy. They claim data from natural experiments and administrative data sets are always spurious and cannot be used to evaluate health policies and other population-wide phenomena in the real world. While many acknowledge biases caused by poor study designs, in this article we argue that several valid designs using administrative data can produce strong findings, particularly the interrupted time series (ITS) design. Many policy studies neither permit nor require an RCT for cause-and-effect inference. Framing our arguments using Campbell and Stanley's classic research design monograph, we show that several "quasi-experimental" designs, especially interrupted time series (ITS), can estimate valid effects (or non-effects) of health interventions and policies as diverse as public insurance coverage, speed limits, hospital safety programs, drug abuse regulation and withdrawal of drugs from the market. We further note the recent rapid uptake of ITS and argue for expanded training in quasi-experimental designs in medical and graduate schools and in post-doctoral curricula.

Keywords: health interventions; quasi-experimental design; randomization; research design.

PubMed Disclaimer

Conflict of interest statement

Compliance with ethical standards Funders This project was supported by a Developmental Research Design grant (Dr. Soumerai and Ms. Ceccarelli) from the Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute. Dr. Soumerai received grant support from the Centers for Disease Control and Prevention’s Natural Experiments for Translation in Diabetes (NEXT-D). Dr. Koppel’s work was in part supported by the Intel-NSF Partnership for Cyber-Physical Systems Security and Privacy. Conflict of interest The authors declare that they do not have a conflict of interest.

Figures

Figure 1
Figure 1
Times series effects of changes in drug benefit limits and cost sharing on the average number of constant-size prescriptions per continuously eligible patient per month among noninstitutionalized New Hampshire patients receiving multiple drugs (n = 860) and other outpatients (n = 8002)
Figure 2
Figure 2
Example of a strong time-series design that controlled for history bias in the Institute for Healthcare Improvement’s (IHI) 100,000 lives campaign. Exhibit is based on data from the Agency for Healthcare Research and Quality (HCUP, 2015).
Figure 3
Figure 3
Upper graph shows fatal and injurious crashes on Arizona interstate highways with the increase to 65 MPH maximum speed limit. The lower graph indicates fatal and injurious crashes on Arizona interstate highways with no change in the 55 MPH maximum speed limit
See this image and copyright information in PMC

References

    1. Ioannidis JP. Are mortality differences detected by administrative data reliable and actionable? JAMA. 2013;309(13):1410–1. doi: 10.1001/jama.2013.3150. - DOI - PubMed
    1. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. - DOI - PMC - PubMed
    1. Lehrer J. The Truth Wears Off. The New Yorker; 2010. http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off. Accessed June 14, 2016.
    1. Freedman DH. Lies, damned lies and medical science. Atlantic. 2010.
    1. Shadish W, Cook T, Campbell D. Experimental and quasi-experimental designs for generalized causal inference. Belmont: Wadsworth Cengage Learning; 2002.

MeSH terms

LinkOut - more resources