Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Sep 3;7(9):e2436535.
doi: 10.1001/jamanetworkopen.2024.36535.

Calibrating Observational Health Record Data Against a Randomized Trial

David Merola  1 , Ulka Campbell  1 , David Lenis  1 , Sebastian Schneeweiss  2 , Shirley Wang  2 , Ann Madsen  1 , Gillis Carrigan  3 , Victoria Chia  3 , Osayi E Ovbiosa  4 , Simone Pinheiro  4 , Nelson Pace  4 , Amanda Bruno  5 , Mark Stewart  6 , Sajan Khosla  7 , Yiduo Zhang  7 , Mothaffar Rimawi  8 , Rachele Hendricks-Sturrup  9 , Jenny Huang  10 , Aliki Taylor  10 , XiaoLong Jiao  11 , Lauren Becnel  11 , Lynn McRoy  11 , Joy Eckert  12 , Carla Rodriguez  12 , Orsolya Lunacsek  13 , Raymond Harvey  14 , Joel Greshock  14 , Khaled Sarsour  14 , Andrew Belli  15 , C K Wang  15 , Laura Fernandes  15 , James Chen  16 , Brian San Francisco  16 , Chithra Sangli  16 , Yana Natanzon  17 , K Arnold Chan  18 , Neil Dhopeshwarkar  18 , Mark Shapiro  19 , Asher Wasserman  19 , Jameson Quinn  19 , Megan Rees  20 , Travis Robinson  20 , Ben Taylor  1 , Jennifer R Rider  1
Affiliations

Calibrating Observational Health Record Data Against a Randomized Trial

David Merola et al. JAMA Netw Open. .

Abstract

Importance: The conditions required for health record data sources to accurately assess treatment effectiveness remain unclear. Emulation of randomized clinical trials (RCTs) with health record data and subsequent calibration of the results can help elucidate this.

Objective: To pilot an emulation of the KEYNOTE-189 RCT using a commercially available electronic health record (EHR) data source.

Design, setting, and participants: This retrospective cohort study used an EHR database spanning from April 2007 to February 2023. Follow-up began on treatment initiation and proceeded until an outcome event, loss to follow-up, end of data, or end of study period (640 days). The population-based cohort was ascertained from EHRs provided by 52 health systems across the US. Eligibility criteria were defined as closely as possible to the benchmark RCT. Patients with non-small cell lung cancer initiating first-line treatment for metastatic disease were included. Patients with evidence of squamous non-small cell lung cancer, primary nonlung malignant neoplasms, or identified EGFR/ALK variations were excluded. Data were analyzed from June to October 2023.

Exposures: Initiation of first-line pembrolizumab and chemotherapy and chemotherapy alone. Chemotherapy in both groups was defined as a combination of pemetrexed and platinum-based (carboplatin or cisplatin) therapy.

Main outcomes and measures: Outcomes of interest were 12-month survival probability and mortality hazard ratio (HR).

Results: A total of 1854 patients (mean [SD] age, 63.7 [9.6] years; 971 [52.4%] men) were eligible, including 589 patients who initiated pembrolizumab and chemotherapy and 1265 patients who initiated chemotherapy only. The cohort included 364 Black patients (19.6%) and 1445 White patients (77.9%). The 12-month survival probabilities were 0.60 (95% CI, 0.54-0.65) in the pembrolizumab group and 0.58 (95% CI, 0.55-0.62) in the chemotherapy-only group, compared with 0.69 (95% CI, 0.64-0.74) in the KEYNOTE-189 pembrolizumab group and 0.49 (95% CI, 0.42-0.56) in the KEYNOTE-189 chemotherapy-only group. The mortality HR was 0.95 (95% CI, 0.78-1.16), compared with 0.49 (95% CI, 0.38-0.64) in the KEYNOTE-189 RCT.

Conclusions and relevance: In this cohort study piloting an RCT emulation, results were incongruous with the benchmark trial. Differences in patient treatment and data capture between the RCT and EHR populations, confounding by indication, treatment crossover, and accuracy of captured diagnoses may explain these findings. Future feasibility assessments will require data sources to have important oncology-specific measures curated.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Merola reported being employed by Aetion during the conduct of the study and being employed by Eli Lilly and Company outside the submitted work. Dr Campbell reported being employed by and owning stock in Aetion outside the submitted work. Dr Lenis reported being employed by and owning stock in Aetion outside the submitted work. Dr Schneeweiss reported owning stock in Aetion and receiving grants from UCB Pharma, Takeda, and Boehringer Ingelheim outside the submitted work. Dr S. Wang reported receiving personal fees from Veracity Healthcare Analytics, Exponent, and MITRE during the conduct of the study. Dr Madsen reported being employed by Aetion during the conduct of the study and having previously been employed by and owning stock in Pfizer outside the submitted work. Dr Carrigan reported being employed by and owning stock in Amgen outside the submitted work. Dr Chia reported being employed by and owning stock in Amgen outside the submitted work. Dr Pinheiro reported being employed by and owning stock in AbbVie during the conduct of the study. Dr Khosla reported being employed by AstraZeneca during the conduct of the study. Dr Rimawi reported receiving personal fees from Pfizer, AstraZeneca, Novartis, and Sermonix and grants from Greenwich LifeSciences (paid to institution) outside the submitted work. Dr A. Taylor reported being employed by and owning stock in Gilead during the conduct of the study. Dr Jiao reported being employed by Bayer during the conduct of the study and having previously been employed by Pfizer outside the submitted work. Dr Becnel reported being employed by and owning stock in Pfizer during the conduct of the study. Dr McRoy reported being employed by Pfizer during the conduct of the study. Dr Lunacsek reported being employed by Bayer during the conduct of the study. Dr Harvey reported being employed by and owning stock in Janssen during the conduct of the study. Dr Greshock reported being employed by Johnson & Johnson during the conduct of the study. Dr Sarsour reported being employed by Johnson & Johnson during the conduct of the study. Dr Belli reported being employed by and owning stock in COTA during the conduct of the study. Dr C.K. Wang reported being employed by COTA during the conduct of the study. Dr Chen reported receiving personal fees from Tempus AI during the conduct of the study. Dr Sangli reported being employed by Tempus AI during the conduct of the study. Dr Natanzon reported being employed by ConcertAI during the conduct of the study. Dr Chan reported being employed by TriNetX during the conduct of the study and receiving grants Amgen and Boehringer Ingelheim outside the submitted work. Dr Dhopeshwarkar reported being employed by and owning stock in TriNetX during the conduct of the study. Dr Wasserman reported owning a patent for US20230177370A1 pending outside the submitted work. Dr B. Taylor reported being employed by Aetion during the conduct of the study and having previously been employed by and owning stock in Pfizer outside the submitted work. Dr Rider reported being employed by Aetion during the conduct of the study and receiving personal fees from Bayer and Monsanto outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Example Patient Timelines for the Primary (Intention-to-Treat) and Secondary (Per-Protocol) Analysis
A, All patients shown are distinct and were classified on the basis of their treatment initiation within a 14-day exposure regimen ascertainment window that is incident with respect to each patient’s first record of metastasis. Notably, this analysis is agnostic to adherence and patients were followed-up from the end of the exposure regimen ascertainment window until occurrence of an outcome or censor event. Additionally, multiple records for each study drug could occur within the ascertainment window, as observed in patients D and H. B, Patients 1 to 4 are cloned and assigned to each treatment strategy on initiation of a study drug. Patient 1 adhered perfectly to the exposure treatment strategy and experienced the outcome event; therefore, all person-time observations were contributed to the exposed group. Patient 1 also contributed a small amount of person-time to the comparator group, as the patient’s treatment history was aligned with the comparator treatment strategy for some follow-up time prior to initiation of D3. Patient 2 in the exposed group was censored on initiation of a cancer treatment that is not a part of their assigned regimen (Dx) and did not contribute person-time to the comparator group, since D3 is not a component of the comparator regimen. Patient 3 in the exposed group was censored at the end of the 21-day interval because that is the point at which the patient’s treatment history was no longer adherent with the exposure strategy. Patient 4 in the comparator group had a treatment history that was compatible with the comparator strategy and experienced an outcome event but is censored in the exposed group due to a lack of adherence with the exposed treatment strategy. Notably, if an event occurred during an interval of follow-up time that was compatible with both treatment strategies, the outcome was attributed to both groups.
Figure 2.
Figure 2.. Attrition Flowchart Describing Study Eligibility of Patients in the Data Source
See this image and copyright information in PMC

References

    1. Lévesque LE, Hanley JA, Kezouh A, Suissa S. Problem of immortal time bias in cohort studies: example using statins for preventing progression of diabetes. BMJ. 2010;340:b5087. doi:10.1136/bmj.b5087 - DOI - PubMed
    1. Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70-75. doi:10.1016/j.jclinepi.2016.04.014 - DOI - PMC - PubMed
    1. Gatto NM, Campbell UB, Rubinstein E, et al. . The Structured Process to Identify Fit-For-Purpose Data: A Data Feasibility Assessment Framework. Clin Pharmacol Ther. 2022;111(1):122-134. doi:10.1002/cpt.2466 - DOI - PMC - PubMed
    1. Gatto NM, Reynolds RF, Campbell UB. A structured preapproval and postapproval comparative study design framework to generate valid and transparent real-world evidence for regulatory decisions. Clin Pharmacol Ther. 2019;106(1):103-115. doi:10.1002/cpt.1480 - DOI - PMC - PubMed
    1. Gatto NM, Campbell UB, Rubinstein E, et al. . The structured process to identify fit-for-purpose data: a data feasibility assessment framework. Clin Pharmacol Ther. 2022;111(1):122-134. doi:10.1002/cpt.2466 - DOI - PMC - PubMed

Publication types

Substances