The risk of false-positive results in orthopaedic surgical trials

Abstract

The risk of concluding that the results of a particular study are true, when, in fact, they really are attributable to chance (or random sampling error) is underappreciated by investigators. This erroneous false-positive conclusion is designated as a Type I or alpha error. The extent to which randomized trials in surgery risk Type I errors is unclear. The current authors hand-searched four orthopaedic journals, six general surgery journals, and five medical journals to identify recently published randomized trials (within the past 2 years). Information on outcomes and statistical adjustment for multiple outcomes was recorded for each study. The risk of a Type I error was calculated for each study that did not explicitly state a primary outcome measure for the main statistical comparison. One hundred fifty-nine studies met the inclusion criteria for the study: 60 studies from orthopaedic journals, 49 studies from nonorthopaedic surgical journals, and 50 studies from medical journals. Of the trials that did not state a primary outcome measure, the risk of Type I errors (false-positive results) in orthopaedic and nonorthopaedic surgery journals (mean 37.3% +/- 13.3% and 37.6% +/- 10.5%, respectively) were significantly greater than medical journals (10.1% +/- 1.9%). In the current review of randomized trials in surgery and medicine, the following is reported: (1) reporting of primary outcomes in trials was inadequate; (2) one in three trials in surgery and one in 10 trials in medicine risked false-positive results; and (3) few trials in surgery and medicine considered adjustment for multiple comparisons.