Threats to applicability of randomised trials: exclusions and selective participation

Abstract

Background: Although the randomised controlled trial (RCT) is regarded as the 'gold standard' in terms of evaluating the effectiveness of interventions, it is susceptible to challenges to its external validity if those participating are unrepresentative of the reference population for whom the intervention in question is intended. In the past, reporting on numbers and types of potential subjects that have been excluded by design, and centres, clinicians or patients that have elected not to participate, has generally been poor, and the threat to inference posed by possible selection bias is unclear.

Methods: A systematic review was undertaken, based largely on MEDLINE and EMBASE with follow-up of cited references, to assess the extent, nature and importance of excluding potential subjects or the unwillingness of particular centres, clinicians or patients to participate.

Results: RCTs vary widely in the extent to which potential future recipients of treatment are included. The reasons cited for excluding certain categories of patient may be medical or scientific. Medical reasons include a high risk of adverse effects and the belief that benefit will be relatively small or absent (or has already been established) in the groups in question. Scientific reasons include more precise estimates of treatment effect because of a relatively homogeneous sample and the reduction of potential bias by excluding those individuals most likely to be lost to follow-up. Many RCTs have blanket exclusions, such as the elderly, women and ethnic minorities, but reasons for these exclusions are seldom given. Evaluative research is undertaken predominantly in university or teaching centres. Non-randomised studies are more likely than RCTs to include non-teaching centres. The effect of patient non-participation appears to depend on whether the RCT is concerned with treatment of an existing condition or with disease prevention. Participants in treatment trials tend to be more severely ill than those who do not participate. In contrast, those who participate in prevention trials are more likely to have adopted a healthy lifestyle than those who decline. Most evaluative studies fail to document adequately the characteristics of those who, while eligible, do not participate. However, subjects included in RCTs (i.e. eligible and participating) tend to have a different prognosis than patients identified from clinical databases.

Conclusions: Narrow inclusion criteria may offer benefits such as increased precision and reduced loss to follow-up, but there are important disadvantages, such as uncertainty about extrapolation of results, which may result in denial of effective treatment to groups who might benefit, and delay in obtaining definitive results because of reduced recruitment rate. Selective participation by teaching centres and sicker patients in treatment RCTs may exaggerate the measured treatment effect. Prevention trials, on the other hand, may underestimate effects as participants have less capacity to benefit.