Computer-aided assessment of the generalizability of clinical trial results

Abstract

Background: The effects of an intervention on patients from populations other than that included in a trial may vary as a result of differences in population features, treatment administration, or general setting. Determining the generalizability of a trial to a target population is important in clinical decision making at both the individual practitioner and policy-making levels. However, awareness to the challenges associated with the assessment of generalizability of trials is low and tools to facilitate such assessment are lacking.

Methods: We review the main factors affecting the generalizability of a clinical trial results beyond the trial population. We then propose a framework for a standardized evaluation of parameters relevant to determining the external validity of clinical trials to produce a "generalizability score". We then apply this framework to populations of patients with heart failure included in trials, cohorts and registries to demonstrate the use of the generalizability score and its graphic representation along three dimensions: participants' demographics, their clinical profile and intervention setting. We use the generalizability score to compare a single trial to multiple "target" clinical scenarios. Additionally, we present the generalizability score of several studies with regard to a single "target" population.

Results: Similarity indices vary considerably between trials and target population, but inconsistent reporting of participant characteristics limit head-to-head comparisons.

Conclusion: We discuss the challenges involved in performing automatic assessment of trial generalizability at scale and propose the adoption of a standard format for reporting the characteristics of trial participants to enable better interpretation of their results.

Keywords: Clinical trials; Decision support; External validity; Generalizability; Similarity assessment.