Few systematic reviews exist documenting the extent of bias: a systematic review

Abstract

Objective: To summarize the evidence concerning bias and confounding in conducting systematic reviews (SRs).

Study design and setting: Literature was identified through searching the Cochrane Library, MEDLINE, PsycINFO until November 2006, and the authors' files. Studies were included if they were SRs of bias that can occur while conducting a SR. Risk of bias in the SRs was appraised using the Oxman and Guyatt index.

Results: Ten SRs were included. All examined biases related to searching for evidence (e.g., publication bias). One also reported bias associated with obtaining data from included studies (e.g., outcome reporting bias). To minimize bias, data suggest including unpublished material, hand searching for additional material, searching multiple databases, assessing for publication bias, and periodically updating SRs. No SRs were found examining biases related to choosing studies for inclusion or combining studies.

Conclusions: There is little evidence from SRs to support commonly practiced methods for conducting SRs. No SRs summarized studies with prospective designs and most had moderate or minimal risk of bias. Future research should examine bias that can occur during the selection of studies for inclusion and the synthesis of studies, as well as systematically review the existing empirical evidence.