Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey

Abstract

Background: Studies that use routinely collected health data (RCD studies) are advocated to complement evidence from randomized controlled trials (RCTs) for comparative effectiveness research and to inform health care decisions when RCTs would be unfeasible. We aimed to evaluate the current use of routinely collected health data to complement RCT evidence.

Methods: We searched PubMed for RCD studies published to 2010 that evaluated the comparative effectiveness of medical treatments on mortality using propensity scores. We identified RCTs of the same treatment comparisons and evaluated how frequently the RCD studies analyzed treatments that had not been compared previously in randomized trials. When RCTs did exist, we noted the claimed motivations for each RCD study. We also analyzed the citation impact of the RCD studies.

Results: Of 337 eligible RCD studies identified, 231 (68.5%) analyzed treatments that had already been compared in RCTs. The study investigators rarely claimed that it would be unethical (6/337) or difficult (18/337) to perform RCTs on the same question. Evidence from RCTs was mentioned or cited by authors of 213 RCD studies. The most common motivations for conducting the RCD studies were alleged limited generalizability of trial results to the "real world" (37.6%), evaluation of specific outcomes (31.9%) or specific populations (23.5%), and inconclusive or inconsistent evidence from randomized trials (25.8%). Studies evaluating "real world" effects had the lowest citation impact.

Interpretation: Most of the RCD studies we identified explored comparative treatment effects that had already been investigated in RCTs. The objective of such studies needs to shift more toward answering pivotal questions that are not supported by trial evidence or for which RCTs would be unfeasible.

Figure 1

Selection of RCD studies for the analysis. RCD = routinely collected health data.

Figure 2

The most frequent motivations for performing observational studies using routinely collected health data (RCD studies) claimed by authors who were aware of existing RCT evidence. Each circular area corresponds to one research motivation category; numbers of studies with multiple motivations are depicted in the corresponding overlapping areas (e.g., 11 RCD studies [5% of 213] reported both the assessment of a specific patient population and the study of "real world" effects as motivations). The diagram is schematic; percentages do not correspond to the size of circular areas.