Evaluating eligibility criteria of oncology trials using real-world data and AI

Abstract

There is a growing focus on making clinical trials more inclusive but the design of trial eligibility criteria remains challenging^1-3. Here we systematically evaluate the effect of different eligibility criteria on cancer trial populations and outcomes with real-world data using the computational framework of Trial Pathfinder. We apply Trial Pathfinder to emulate completed trials of advanced non-small-cell lung cancer using data from a nationwide database of electronic health records comprising 61,094 patients with advanced non-small-cell lung cancer. Our analyses reveal that many common criteria, including exclusions based on several laboratory values, had a minimal effect on the trial hazard ratios. When we used a data-driven approach to broaden restrictive criteria, the pool of eligible patients more than doubled on average and the hazard ratio of the overall survival decreased by an average of 0.05. This suggests that many patients who were not eligible under the original trial criteria could potentially benefit from the treatments. We further support our findings through analyses of other types of cancer and patient-safety data from diverse clinical trials. Our data-driven methodology for evaluating eligibility criteria can facilitate the design of more-inclusive trials while maintaining safeguards for patient safety.

Extended Data Fig. 1 |. Selection of aNSCLC clinical trials.

Workflow implemented in a Python script to perform a systematic selection of trials using the six filters described in the Methods. Twenty clinical trials met the first five filters, but only six of them had a protocol that was publicly available either on ClinicalTrials.gov or as supplementary material in the associated publications. Additionally, four trials were included in the model that were suggested by subject matter experts at Roche. These four trials had not originally been identified by our systematic search owing to errors in their clinicaltrials.gov entries (for example, one trial was listed as having eight arms despite having only two).

Extended Data Fig. 2 |. Differential use of eligibility criteria.

The trial and criteria grid shows which eligibility criteria are present in each aNSCLC trial (criteria coloured in yellow are included in the trial protocol). The trials are divided into first-line and second-line therapies, depending on their protocol design; the eligibility criteria are grouped into categories depending on the type of variable that is measured.

Extended Data Fig. 3 |. Balance assessment for treatment and control groups.

a–j, For each aNSCLC trial, we plot the standardized mean difference (SMD) for every patient covariate between the treatment and control cohorts generated from the Flatiron data. SMD values close to 0 indicate that the cohorts are balanced. The inverse propensity weighting used in our analysis (IPTW) effectively balances the cohort. ‘Raw’ corresponds to the unadjusted cohorts.

Extended Data Fig. 4 |. Convergence of the Shapley value for the bilirubin criterion.

The x axis indicates the number of randomly generated subsets of criteria used for Shapley value computation.

Extended Data Fig. 5 |. Example of the effect of relaxing the eligibility criteria.

a–c, Survival curves, hazard ratios and the number of patients in trial Keynote189 when the eligibility criteria scenarios are: the original trial criteria (a), fully relaxed criteria (that is, all of the patients who took the relevant treatments) (b) and the data-driven criteria identified by Trial Pathfinder (c).

Extended Data Fig. 6 |. Comparison of patient baselines.

a–e, Violin plots for the laboratory values of the patients at the start of treatment. We partition the sampled patients with aNSCLC from the Flatiron database into two groups depending on whether or not they had withdrawn from first-line aNSCLC treatments due to toxicity (82 patients with toxicity = true and 918 patients with toxicity = false). The violin plots show the distribution of each of the laboratory values at the start of the trial. There is no significant difference in the baseline laboratory values between patients who later withdrew from treatment due to toxicity and the patients who did not (unadjusted two-sided Student’s t-test; P > 0.2 for all five laboratory tests).

Extended Data Fig. 7 |. Effects of varying laboratory cut-off values.

a–d, Changes in the Shapley value of the hazard ratios of the overall survival for different laboratory values thresholds. The x axis corresponds to different values of the inclusion threshold for bilirubin (serum bilirubin less than threshold for inclusion) (a), platelets (platelet count larger than the threshold) (b), haemoglobin (whole-blood haemoglobin level less than the threshold) (c) and ALP (ALP concentration larger than the threshold) (d). Changing a threshold to the right on the x axis corresponds to more relaxed criteria that would include more patients. The thresholds used in the original trials are provided in the key and their Shapley values are set as the baseline 0. For most of the trials, relaxing the laboratory value thresholds would not significantly change the hazard ratio or would decrease the hazard ratio (that is, curve below 0). The range of values shown for each laboratory test corresponds to the range of thresholds used in actual trials (Supplementary Table 35). In all of the panels, the error bars correspond to the bootstrap standard deviation and the centres correspond to the bootstrap mean of five replications.

Fig. 1 |. Trial Pathfinder workflow and applications.

a, Trial Pathfinder takes as input the real-world dataset and the target trial protocol (treatments and eligibility criteria). It programmatically encodes the eligibility criteria and performs trial emulation using propensity score weighting. It then performs a survival analysis on the emulated treatment groups, and reports both the number of eligible patients and the resulting hazard ratio. b, Combining an importance analysis of the automated criteria with the Shapley value, Trial Pathfinder evaluates individual criteria and derives a data-driven set of criteria that expands the pool of eligible patients without reducing the effect size. This can guide the design of trials. ALK, anaplastic lymphoma kinase; ECOG, Eastern Cooperative Oncology Group; HR, hazard ratio; IPTW, inverse probability of treatment weighting; PDL1, programmed death ligand 1; RWD, real-world data.

Fig. 2 |. Influences of individual eligibility rules.

a, b, Shapley values of the hazard ratio of overall survival (a) and changes in the number of eligible patients (b) are shown across different aNSCLC trials and eligibility criteria. a, Red, inclusion of the criterion increases the hazard ratio; blue, the criterion decreases the hazard ratio when included, on average. b, The fraction of patients who would be excluded by each criterion in every trial is shown. 1L, first line of therapy; 2L, second line of therapy; CNS, central nervous system; Pt, platinum; WBC, white blood cell count. The ‘CNS metastasis exclude’ criterion means that patients with CNS metastases are excluded.