Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples

Sara Lodi¹, Andrew Phillips², Jens Lundgren³, Roger Logan⁴, Shweta Sharma⁵, Stephen R Cole, Abdel Babiker⁶, Matthew Law⁷, Haitao Chu⁵, Dana Byrne⁸, Andrzej Horban⁹, Jonathan A C Sterne^{10

11}, Kholoud Porter², Caroline Sabin², Dominique Costagliola¹², Sophie Abgrall^{12

13}, John Gill^{14

15}, Giota Touloumi¹⁶, Antonio G Pacheco¹⁷, Ard van Sighem¹⁸, Peter Reiss^{18

19

20}, Heiner C Bucher²¹, Alexandra Montoliu Giménez²², Inmaculada Jarrin²³, Linda Wittkop²⁴, Laurence Meyer^{25

26}, Santiago Perez-Hoyos²⁷, Amy Justice²⁸, James D Neaton⁵, Miguel A Hernán^{4

29

30}; INSIGHT START Study Group and the HIV-CAUSAL Collaboration

Affiliations

¹ Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts.
² Institute for Global Health, University College London, United Kingdom.
³ Department of Infectious Diseases, Rigshospitalet, University of Copenhagen, Denmark.
⁴ Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts.
⁵ Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota.
⁶ Medical Research Council, Clinical Trials Unit in University College London, London, United Kingdom.
⁷ The Kirby Institute, Sidney, Australia.
⁸ Division of Infectious Diseases, Department of Medicine, Cooper University Hospital, Cooper Medical School at Rowan University, New Jersey.
⁹ Medical University of Warsaw, Department for Adult's Infectious Diseases, Warsaw, Poland.
¹⁰ Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, North Carolina.
¹¹ Department of Population Health Sciences, University of Bristol, Bristol, United Kingdom.
¹² INSERM, Sorbonne Université, Institut Pierre Louis d'Épidémiologie et de Santé Publique (IPLESP), Paris, France.
¹³ AP-HP, Hôpital Antoine Béclère, Service de Médecine Interne, Clamart, France.
¹⁴ Southern Alberta Clinic, Calgary, Canada.
¹⁵ Department of Medicine, University of Calgary, Canada.
¹⁶ National and Kapodistrian University of Athens, Faculty of Medicine, Dept. of Hygiene, Epidemiology and Medical Statistics, Greece.
¹⁷ Programa de Computação Científica, Fundacao Oswaldo Cruz, Rio de Janeiro, Brasil.
¹⁸ Stichting HIV Monitoring, Amsterdam, the Netherlands.
¹⁹ Amsterdam University Medical Centres, University of Amsterdam, Department of Global Health and Division of Infectious Diseases, Amsterdam, the Netherlands.
²⁰ Amsterdam Institute for Global Health and Development, and Amsterdam Public Health Research Institute, Amsterdam, the Netherlands.
²¹ Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, Switzerland.
²² Centre for Epidemiological Studies on HIV/STI in Catalonia (CEEISCAT), Agència de Salut Pública de Catalunya (ASPC), Badalona, Spain.
²³ Centro Nacional de Epidemiología, Instituto de Salud Carlos III, Madrid, Spain.
²⁴ Univ. Bordeaux, ISPED, Inserm, Bordeaux Population Health Research Center, team MORPH3EUS, UMR 1219, CIC-EC 1401, Bordeaux, France.
²⁵ CHU de Bordeaux, Pôle de santé publique, Service d'information médicale, Bordeaux, France.
²⁶ Université Paris Sud, UMR 1018, le Kremlin Bicêtre, France.
²⁷ Vall d'Hebrón Research Institute; Barcelona, Spain.
²⁸ Yale University School of Medicine, New Haven, Connecticut.
²⁹ Department of Biostatistics, Harvard T.H. Chan School of Public Health.
³⁰ Harvard-MIT Division of Health Sciences and Technology, Boston, Massachusetts.

PMID: 31063192
PMCID: PMC6670045
DOI: 10.1093/aje/kwz100

Comparative Study

Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples

Sara Lodi et al. Am J Epidemiol. 2019.

. 2019 Aug 1;188(8):1569-1577.

doi: 10.1093/aje/kwz100.

Authors

Affiliations

¹ Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts.
² Institute for Global Health, University College London, United Kingdom.
³ Department of Infectious Diseases, Rigshospitalet, University of Copenhagen, Denmark.
⁴ Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts.
⁵ Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota.
⁶ Medical Research Council, Clinical Trials Unit in University College London, London, United Kingdom.
⁷ The Kirby Institute, Sidney, Australia.
⁸ Division of Infectious Diseases, Department of Medicine, Cooper University Hospital, Cooper Medical School at Rowan University, New Jersey.
⁹ Medical University of Warsaw, Department for Adult's Infectious Diseases, Warsaw, Poland.
¹⁰ Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, North Carolina.
¹¹ Department of Population Health Sciences, University of Bristol, Bristol, United Kingdom.
¹² INSERM, Sorbonne Université, Institut Pierre Louis d'Épidémiologie et de Santé Publique (IPLESP), Paris, France.
¹³ AP-HP, Hôpital Antoine Béclère, Service de Médecine Interne, Clamart, France.
¹⁴ Southern Alberta Clinic, Calgary, Canada.
¹⁵ Department of Medicine, University of Calgary, Canada.
¹⁶ National and Kapodistrian University of Athens, Faculty of Medicine, Dept. of Hygiene, Epidemiology and Medical Statistics, Greece.
¹⁷ Programa de Computação Científica, Fundacao Oswaldo Cruz, Rio de Janeiro, Brasil.
¹⁸ Stichting HIV Monitoring, Amsterdam, the Netherlands.
¹⁹ Amsterdam University Medical Centres, University of Amsterdam, Department of Global Health and Division of Infectious Diseases, Amsterdam, the Netherlands.
²⁰ Amsterdam Institute for Global Health and Development, and Amsterdam Public Health Research Institute, Amsterdam, the Netherlands.
²¹ Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, Switzerland.
²² Centre for Epidemiological Studies on HIV/STI in Catalonia (CEEISCAT), Agència de Salut Pública de Catalunya (ASPC), Badalona, Spain.
²³ Centro Nacional de Epidemiología, Instituto de Salud Carlos III, Madrid, Spain.
²⁴ Univ. Bordeaux, ISPED, Inserm, Bordeaux Population Health Research Center, team MORPH3EUS, UMR 1219, CIC-EC 1401, Bordeaux, France.
²⁵ CHU de Bordeaux, Pôle de santé publique, Service d'information médicale, Bordeaux, France.
²⁶ Université Paris Sud, UMR 1018, le Kremlin Bicêtre, France.
²⁷ Vall d'Hebrón Research Institute; Barcelona, Spain.
²⁸ Yale University School of Medicine, New Haven, Connecticut.
²⁹ Department of Biostatistics, Harvard T.H. Chan School of Public Health.
³⁰ Harvard-MIT Division of Health Sciences and Technology, Boston, Massachusetts.

PMID: 31063192
PMCID: PMC6670045
DOI: 10.1093/aje/kwz100

Abstract

Effect estimates from randomized trials and observational studies might not be directly comparable because of differences in study design, other than randomization, and in data analysis. We propose a 3-step procedure to facilitate meaningful comparisons of effect estimates from randomized trials and observational studies: 1) harmonization of the study protocols (eligibility criteria, treatment strategies, outcome, start and end of follow-up, causal contrast) so that the studies target the same causal effect, 2) harmonization of the data analysis to estimate the causal effect, and 3) sensitivity analyses to investigate the impact of discrepancies that could not be accounted for in the harmonization process. To illustrate our approach, we compared estimates of the effect of immediate with deferred initiation of antiretroviral therapy in individuals positive for the human immunodeficiency virus from the Strategic Timing of Antiretroviral Therapy (START) randomized trial and the observational HIV-CAUSAL Collaboration.

Keywords: antiretroviral initiation; causal inference; per-protocol effect; target trial.

© The Author(s) 2019. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.

PubMed Disclaimer

Comment in

RE: "EFFECT ESTIMATES IN RANDOMIZED TRIALS AND OBSERVATIONAL STUDIES: COMPARING APPLES WITH APPLES".
Schomaker M, Kühne F, Siebert U. Schomaker M, et al. Am J Epidemiol. 2020 Jan 31;189(1):77-78. doi: 10.1093/aje/kwz194. Am J Epidemiol. 2020. PMID: 31529036 No abstract available.

References

1. Hernán MA, Alonso A, Logan R, et al. . Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19(6):766–779. - PMC - PubMed
1. Hernán MA, Sauer BC, Hernández-Díaz S, et al. . Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70–75. - PMC - PubMed
1. Hernán MA, Robins JM. Per-protocol analyses of pragmatic trials. N Engl J Med. 2017;377(14):1391–1398. - PubMed
1. Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2014;(4):MR000034. - PMC - PubMed
1. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–1886. - PubMed

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples

Affiliations

Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples

Authors

Affiliations

Abstract

Comment in

References

Publication types

MeSH terms

Substances

Grants and funding

LinkOut - more resources

Full Text Sources

Medical