Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jun 16;4(2):ooab045.
doi: 10.1093/jamiaopen/ooab045. eCollection 2021 Apr.

Illustrating potential effects of alternate control populations on real-world evidence-based statistical analyses

Yidi Huang  1 , William Yuan  1 , Isaac S Kohane  1 , Brett K Beaulieu-Jones  1
Affiliations

Illustrating potential effects of alternate control populations on real-world evidence-based statistical analyses

Yidi Huang et al. JAMIA Open. .

Abstract

Objective: Case-control study designs are commonly used in retrospective analyses of real-world evidence (RWE). Due to the increasingly wide availability of RWE, it can be difficult to determine whether findings are robust or the result of testing multiple hypotheses.

Materials and methods: We investigate the potential effects of modifying cohort definitions in a case-control association study between depression and type 2 diabetes mellitus. We used a large (>75 million individuals) de-identified administrative claims database to observe the effects of minor changes to the requirements of glucose and hemoglobin A1c tests in the control group.

Results: We found that small permutations to the criteria used to define the control population result in significant shifts in both the demographic structure of the identified cohort as well as the odds ratio of association. These differences remain present when testing against age- and sex-matched controls.

Discussion: Analyses of RWE need to be carefully designed to avoid issues of multiple testing. Minor changes to control cohorts can lead to significantly different results and have the potential to alter even prospective studies through selection bias.

Conclusion: We believe this work offers strong support for the need for robust guidelines, best practices, and regulations around the use of observational RWE for clinical or regulatory decision-making.

Keywords: association testing; informatics; phenotyping; real-world evidence.

PubMed Disclaimer

Conflict of interest statement

None declared.

Figures

Figure 1.
Figure 1.
Overview of study design to evaluate the effect of alternate control groups on the association between depression and type 2 diabetes mellitus.
Figure 2.
Figure 2.
Control algorithms for type 2 diabetes mellitus. (A) Baseline controls as defined by eMERGE. (B) Controls where the glucose lab value is ignored. (C) Controls where whether the member has had a glucose lab or not is ignored. (D) Controls where the member has not had a glucose test.
Figure 3.
Figure 3.
Potential selection effects from glucose lab testing requirement. (A) Number of glucose tests per member during 2011 by age. (B) Distributions of HbA1c value from our dataset compared with privately insured individuals from the National Health and Nutrition Examination Survey (NHANES).
Figure 4.
Figure 4.
Age distributions under each different control definition.
Figure 5.
Figure 5.
Odds ratio among each of the different control groups for 200 samples. (A) Sample size of 2000 without matching, (B) sample size of 2000 with matching on age and sex, (C) sample size of 10 000 without matching, and (D) sample size of 10 000 with matching on age and sex.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Jarow JP, LaVange L, Woodcock J.. Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data. JAMA 2017; 318: 703–4. - PubMed
    1. Corrigan-Curay J, Sacks L, Woodcock J.. Real-world evidence and real-world data for evaluating drug safety and effectiveness. JAMA 2018; 320: 867. - PubMed
    1. The FDA and Flatiron Health Expand Real-World Data Cancer Research Collaboration | Flatiron Health. Flatiron Health. 2019. https://flatiron.com/press/press-release/the-fda-and-flatiron-health-exp... Accessed March 4, 2019.
    1. FDA. Number of Breakthrough Therapy Designation Requests Received. FDA Data Access. 2019. https://www.accessdata.fda.gov/scripts/fdatrack/view/track.cfm?program=c... Accessed September 10, 2019.
    1. FDA. CDER Drug and Biologic Accelerated Approvals Based on a Surrogate Endpoint. fda.gov. 2019. https://www.fda.gov/media/88907/download Accessed August 3, 2019.

LinkOut - more resources