Generalizability of findings from randomized controlled trials is limited in the leading general medical journals

Abstract

Objectives: To document reporting of study characteristics of randomized controlled trials (RCTs) in the four leading general medical journals and to appraise the generalizability of the evidence.

Study design and setting: All RCTs in BMJ, JAMA, Lancet, and NEJM from January 1, 2017 to September 30, 2017 were searched by hand, and data were extracted according to the benchmarking method.

Results: Hundred sixty-one RCTs were found; 67% assessed pharmacological therapy. The percentages of adequate documentation were patients' path before randomization 3% to 33% of trials; characteristics of the health care settings 0% to 75%; at least two comorbid conditions 25% to 50%; at least one measure was reported of functioning 42% to 54%, of behavioral factors 25% to 58%, of environmental factors 3% to 25%, and of inequity-related factors 28% to 68%; cointerventions 6% to 25%; and reasons for dropping out of follow-up 39% to 100%.

Conclusion: Almost all RCTs showed deficiencies in description of patient selection and study setting and in reporting of patient characteristics related to functioning, comorbidities, and to behavioral, environmental, and inequity factors. The findings indicate that generalizability of this evidence may be limited. The benchmarking method can be used for planning and appraisal of clinical trials and systematic reviews.

Keywords: Benchmarking method; Generalizability; Medical journals; Randomized controlled trial; Risk of bias; Systematic review.