Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Aug 10;2011(8):MR000026.
doi: 10.1002/14651858.MR000026.pub2.

Checking reference lists to find additional studies for systematic reviews

Tanya Horsley  1 , Orvie Dingwall, Margaret Sampson
Affiliations

Checking reference lists to find additional studies for systematic reviews

Tanya Horsley et al. Cochrane Database Syst Rev. .

Abstract

Background: Checking reference lists to identify relevant studies for systematic reviews is frequently recommended by systematic review manuals and is often undertaken by review authors. To date, no systematic review has explicitly examined the effectiveness of checking reference lists as a method to supplement electronic searching.

Objectives: To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.

Search strategy: We searched the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science abstracts (LISA) (1969 to July 2008) and MEDLINE (1966 to July 2008). We contacted experts in systematic review methods and examined reference lists of articles.

Selection criteria: Studies of any design which examined checking reference lists as a search method for systematic reviews in any area. The primary outcome was the additional yield of relevant studies (i.e. studies not found through any other search methodologies); other outcomes were publication types identified and data pertaining to the costs (e.g. cost-effectiveness, cost-efficiency) of checking reference lists.

Data collection and analysis: We summarized data descriptively.

Main results: We included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias. The additional yield (calculated by dividing the additional 'unique' yield identified by checking reference lists by the total number of studies found to be eligible within the study) of relevant studies identified through checking reference lists ranged from 2.5% to 42.7%. Only two studies reported yield information by publication type (dissertations and systematic reviews). No cost data were reported although one study commented that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper, and for that particular study costs were not specifically estimated.

Authors' conclusions: There is some evidence to support the use of checking reference lists for locating studies in systematic reviews. However, this evidence is derived from weak study designs. In situations where the identification of all relevant studies through handsearching and database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. The challenge, therefore, is for review authors to recognize those situations.

PubMed Disclaimer

Conflict of interest statement

None declared.

Figures

1
1
Study flow diagram.
See this image and copyright information in PMC

Update of

References

References to studies included in this review

Avenell 2001 {published data only}
    1. Avenell A, Handoll HH, Grant AM. Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. American Journal of Clinical Nutrition 2001;73(3):505‐10. - PubMed
Greenhalgh 2005 {published data only}
    1. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005;331(7524):1064‐5. - PMC - PubMed
Kleijnen 1992 {published data only}
    1. Kleijnen J, Knipschild P. The comprehensiveness of MEDLINE and EMBASE computer searches. Pharmaceutisch Weekblad Scientific edition 1992;14(5):316‐20. - PubMed
Lemeshow 2005 {published data only}
    1. Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta‐analysis of observational studies. Journal of Clinical Epidemiology 2005;58(9):867‐73. - PubMed
McManus 1998 {published data only}
    1. McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ 1998;317(7172):1562‐3. - PMC - PubMed
McNally 2004 {published data only}
    1. McNally R, Alborz A. Developing methods for systematic reviewing in health services delivery and organization: an example from a review of access to health care for people with learning disabilities. Part 1. Identifying the literature. Health and Information Libraries Journal 2004;21(3):182‐92. - PMC - PubMed
Murphy 2003 {published data only}
    1. Murphy LS, Reinsch S, Najm WI, Dickerson VM, Seffinger MA, Adams A, et al. Spinal palpation: the challenges of information retrieval using available databases. Journal of Manipulative Physiological Therapeutics 2003;26(6):374‐82. - PubMed
Ogilvie 2005 {published data only}
    1. Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go?. Journal of Epidemiology and Community Health 2005;59(9):804‐8. - PMC - PubMed
Rees 2003 {published data only}
    1. Rees R, Potter S, Penn H. Searching for studies of the outcomes of education: a bibliometric study [abstract]. XI Cochrane Colloquium: Evidence, Health Care and Culture; Oct 26‐31 2003. Barcelona, Spain, 2003:45.
Savoie 2003 {published data only}
    1. Helmer D, Savoie I, Green C, Kazanjian A. Evidence‐based practice: extending the search to find material for the systematic review. Bulletin of the Medical Library Association 2001;89(4):346‐52. - PMC - PubMed
    1. Savoie I, Helmer D, Green CJ, Kazanjian A. Reducing bias through extended systematic review search. International Journal of Technology Assessment in Health Care 2003;191(1):168‐78. - PubMed
Stevinson 2004 {published data only}
    1. Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns?. Complementary Therapies in Medicine 2004;12(4):228‐32. - PubMed
van Loo 1985 {published data only}
    1. Loo J. Medical and psychological effects of unemployment: a 'grey' literature search. Health Libraries Review 1985;2:55‐62.

Additional references

Allison 1999
    1. Allison JJ, Kiefe CI, Weissman NW, Carter J, Centor RM. The art and science of searching MEDLINE to answer clinical questions. Finding the right number of articles. International Journal of Technology Assessment in Health Care 1999;15(2):281‐96. - PubMed
Beck 2010
    1. Beck C. The scoping study in physical therapy: application of traditional systematic review guidelines to an emerging methodology. Canadian Health Library Association Conference. June 2010.
Berry 2000
    1. Berry E, Kelly S, Hutton J, Harris KM, Smith MA. Identifying studies for systematic reviews. An example from medical imaging. International Journal of Technology Assessment in Health Care 2000;16(2):668‐72. - PubMed
Budd 1998
    1. Budd JM, Sievert M, Schultz TR. Phenomena of retraction: reasons for retraction and citations and practice in medicine. JAMA 1998;280(3):296‐7. - PubMed
Budd 1999
    1. Budd JM, Sievert M, Schultz TR. Effects of article retraction on citation and practice in medicine. Bulletin of the Medical Libraries Association 1999;87(4):437‐43. - PMC - PubMed
Centre for Reviews and Dissemination 2008
    1. Centre for Reviews and Dissemination. Systematic reviews: CRD's guidance for undertaking reviews in healthcare [Internet]. Available from: http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf (accessed 16 July 2008).
Cooper 1994
    1. Cooper H, Hedges L. The handbook of research synthesis. New York: Russell Sage Foundation, 1994.
Dickersin 1994
    1. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309(6964):1286‐91. - PMC - PubMed
Harbour 2001
    1. Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ 2001;323(7308):334‐6. - PMC - PubMed
Harbourt 1995
    1. Harbourt AM, Knecht LS, Humphreys BL. Structured abstracts in MEDLINE, 1989‐1991. Bulletin of the Medical Library Association 1995;83(2):190‐5. - PMC - PubMed
Hayward 1997
    1. Hayward S, Brunton G, Thomas K, Ciliska D. Searching for the evidence: source, time and yield. 2nd International Conference Scientific Basis of Health Services & 5th Annual Cochrane Colloquium. Amsterdam, October 1997.
Helmer 2001
    1. Helmer D, Savoie I, Green C, Kazanjian A. Evidence‐based practice: extending the search to find material for the systematic review. Bulletin of the Medical Library Association 2001;89(4):346‐52. - PMC - PubMed
Higgins 2008
    1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0 [updated Feb 2008]. The Cochrane Collaboration available at www.cochrane‐handbook.org 2008.
Hopewell 2007
    1. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database of Systematic Reviews 2007, Issue 1. [DOI: 10.1002/14651858.MR000001.pub2] - DOI - PMC - PubMed
HuGENet 2008
    1. HuGENet. The HuGENet HuGE Review Handbook. Available at: http://www.medicine.uottawa.ca/public‐health‐genomics/web/assets/documen... Review Handbook V1.0.pdf (accessed 18 July 2010).
Kochtanek 1982
    1. Kochtanek TR. Bibliographic compilation using reference and citation links. Information Processing and Management 1982;18(1):33‐9.
Lefebvre 2009
    1. Levebre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]. The Cochrane Collaboration, 2009. Available from www.cochrane‐handbook.org.
May 2007
    1. May C, Mair F, Dowrick C, Finch T. Process evaluation for complex interventions in primary care: understanding trials using the normalization process model. BMC Family Practice 2007;8(1):42. - PMC - PubMed
Medical Research Council 2000
    1. Medical Research Council. A framework for development and evaluation of complex interventions to improve health. http://www.mrc.ac.uk/Utilities/Documentrecord/index.htm?d=MRC003372 2000. - PMC - PubMed
Moher 2007
    1. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007;4(3):e78. - PMC - PubMed
Moher 2009
    1. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred Reporting Items for Systematic Reviews and Meta‐Analyses: The PRISMA Statement. Open Medicine 2009;3(3):123‐30. - PMC - PubMed
Pagliaro 2003
    1. Pagliaro U, Nieri M, Franceschi D, Clauser C, Pini‐Prato G. Evidence‐based mucogingival therapy. Part 1: A critical review of the literature on root coverage procedures. Journal of Periodontology 2003;74(5):709‐40. - PubMed
Pao 1989
    1. Pao ML. Retrieval differences between term and citation indexing. Information knowledge, evolution. (Proceedings of the 4th FID Congress, Helsinki, 28 August ‐ 1 September 1988). Amsterdam: Elsevier Science Publishers, 1989:113‐20.
Pfeifer 1990
    1. Pfeifer MO, Snodgrass GL. The continued use of retracted, invalid scientific literature. JAMA 1990;263(10):1420‐3. - PubMed
Royle 2003
    1. Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. International Journal of Technology Assessment in Health Care 2003;19(4):591‐603. - PubMed
Sackett 1996
    1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ 1996;312(7023):71‐2. - PMC - PubMed
Sampson 2008a
    1. Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting for systematic reviews. Journal of Clinical Epidemiology 2008;61(8):748‐54. - PubMed
Sampson 2008b
    1. Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health 2008.
Stielow 1988
    1. Stielow F, Tibbo H. The negative search, online reference and the humanities: a critical essay in library literature. RQ 1988;27(3):358‐65.
Wager 2008
    1. Wager E, Middleton P. Technical editing of research reports in biomedical journals. Cochrane Database of Systematic Reviews 2008, Issue 4. [DOI: 10.1002/14651858.MR000002.pub3] - DOI - PMC - PubMed

Publication types

MeSH terms

LinkOut - more resources