Can Cochrane Reviews in controversial areas be biased? A sensitivity analysis based on the protocol of a Systematic Cochrane Review on low-level laser therapy in osteoarthritis

Abstract

Objective: The aim of this study was to test if a conclusion in a systematic review of low-level laser therapy (LLLT) for osteoarthritis from the Cochrane Library was valid and robust.

Background data: Health policy decisions often rely on conclusions from the Cochrane Database of Systematic Reviews for approval of new therapies, although their validity for controversial non-pharmacological treatment has been questioned.

Methods: Validity was tested against a nine-item checklist for systematic reviews. Review selections were analyzed for possible discrepancies between trial and review reports, and omissions of relevant trials and data. Alternative data from discrepancies and omissions were then imputed in a sensitivity analysis, to test if review conclusions were robust.

Results: Only clinicians who had performed LLLT trials with negative results were invited into the review group. Review quality was sound in areas of literature search and methodological assessments, and some of the limitations were mentioned. The statistical analysis held 18 questionable selections such as omissions of trials, data, and subgroup analyses. These selections systematically favored the negative review conclusion. Without altering the review protocol, the sensitivity analysis of combined results changed to significantly positive for continuous and categorical data when data from all included trials were combined. Further sensitivity analyses with inclusion of valid non-included trials, performance of missing follow-up, and subgroup analyses revealed consistent and highly significant results in favor of active LLLT.

Conclusions: In this example, the Cochrane review conclusion was neither robust nor valid. Representation of experts and different views on efficacy in the review group and extensive use of sensitivity analyses could probably improve quality control of reviews in areas of controversy.