Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Nov 9;2011(11):MR000027.
doi: 10.1002/14651858.MR000027.pub2.

Methods for obtaining unpublished data

Taryn Young  1 , Sally Hopewell
Affiliations

Methods for obtaining unpublished data

Taryn Young et al. Cochrane Database Syst Rev. .

Abstract

Background: In order to minimise publication bias, authors of systematic reviews often spend considerable time trying to obtain unpublished data. These include data from studies conducted but not published (unpublished data), as either an abstract or full-text paper, as well as missing data (data available to original researchers but not reported) in published abstracts or full-text publications. The effectiveness of different methods used to obtain unpublished or missing data has not been systematically evaluated.

Objectives: To assess the effects of different methods for obtaining unpublished studies (data) and missing data from studies to be included in systematic reviews.

Search methods: We identified primary studies comparing different methods of obtaining unpublished studies (data) or missing data by searching the Cochrane Methodology Register (Issue 1, 2010), MEDLINE and EMBASE (1980 to 28 April 2010). We also checked references in relevant reports and contacted researchers who were known or who were thought likely to have carried out relevant studies. We used the Science Citation Index and PubMed 'related articles' feature to identify any additional studies identified by other sources (19 June 2009).

Selection criteria: Primary studies comparing different methods of obtaining unpublished studies (data) or missing data in the healthcare setting.

Data collection and analysis: The primary outcome measure was the proportion of unpublished studies (data) or missing data obtained, as defined and reported by the authors of the included studies. Two authors independently assessed the search results, extracted data and assessed risk of bias using a standardised data extraction form. We resolved any disagreements by discussion.

Main results: Six studies met the inclusion criteria; two were randomised studies and four were observational comparative studies evaluating different methods for obtaining missing data.Methods to obtain missing dataFive studies, two randomised studies and three observational comparative studies, assessed methods for obtaining missing data (i.e. data available to the original researchers but not reported in the published study).Two studies found that correspondence with study authors by e-mail resulted in the greatest response rate with the fewest attempts and shortest time to respond. The difference between the effect of a single request for missing information (by e-mail or surface mail) versus a multistage approach (pre-notification, request for missing information and active follow-up) was not significant for response rate and completeness of information retrieved (one study). Requests for clarification of methods (one study) resulted in a greater response than requests for missing data. A well-known signatory had no significant effect on the likelihood of authors responding to a request for unpublished information (one study). One study assessed the number of attempts made to obtain missing data and found that the number of items requested did not influence the probability of response. In addition, multiple attempts using the same methods did not increase the likelihood of response. METHODS TO OBTAIN UNPUBLISHED STUDIES: One observational comparative study assessed methods to obtain unpublished studies (i.e. data for studies that have never been published). Identifying unpublished studies ahead of time and then asking the drug industry to provide further specific detail proved to be more fruitful than sending of a non-specific request.

Authors' conclusions: Those carrying out systematic reviews should continue to contact authors for missing data, recognising that this might not always be successful, particularly for older studies. Contacting authors by e-mail results in the greatest response rate with the fewest number of attempts and the shortest time to respond.

PubMed Disclaimer

Conflict of interest statement

None known.

Figures

1
1
Study flow diagram.
See this image and copyright information in PMC

Update of

  • doi: 10.1002/14651858.MR000027

References

References to studies included in this review

Brown 2003 {published data only}
    1. Brown T, Hooper L. Effectiveness of brief contact with authors. XI Cochrane Colloquium: Evidence, Health Care and Culture; 2003 Oct 26‐31; Barcelona, Spain. 2003.
Gibson 2006 {published data only}
    1. Gibson CA, Bailey BW, Carper MJ, Lecheminant JD, Kirk EP, Huang G, et al. Author contacts for retrieval of data for a meta‐analysis on exercise and diet restriction. International Journal of Technology Assessment in Health Care 2006;22(2):267‐70. - PubMed
Guevara 2005 {published data only}
    1. Guevara J, Keren R, Nihtianova S, Zorc J. How do authors respond to written requests for additional information?. XIII Cochrane Colloquium; 2005 Oct 22‐26; Melbourne, Australia. 2005.
Higgins 1999 {published data only}
    1. Higgins J, Soornro M, Roberts I, Clarke M. Collecting unpublished data for systematic reviews: a proposal for a randomised trial. 7th Annual Cochrane Colloquium Abstracts, October 1999 in Rome. 1999.
Milton 2001 {published data only}
    1. Milton J, Logan S, Gilbert R. Well‐known signatory does not affect response to a request for information from authors of clinical trials: a randomised controlled trial. 9th Annual Cochrane Colloquium Abstracts, October 2001 in Lyon. 2001.
Shukla 2003 {published data only}
    1. Shukla V. The challenge of obtaining unpublished information from the drug industry. XI Cochrane Colloquium: Evidence, Health Care and Culture; 2003 Oct 26‐31; Barcelona, Spain. 2003.

References to studies excluded from this review

Bohlius 2003 {published data only}
    1. Bohlius J, Langensiepen S, Engert A. Data hunting: a case report. XI Cochrane Colloquium: Evidence, Health Care and Culture; 2003 Oct 26‐31; Barcelona, Spain. 2003.
Eysenbach 2001 {published data only}
    1. Eysenbach G, Tuische J, Diepgen TL. Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews. Medical Informatics and the Internet in Medicine 2001;26(3):203‐18. - PubMed
    1. Eysenbach G, Tuische J, Diepgen TL. Evaluation of the usefulness of internet searches to identify unpublished clinical trials for systematic reviews. Chinese Journal of Evidence‐Based Medicine 2002;2(3):196‐200. - PubMed
Hadhazy 1999 {published data only}
    1. Hadhazy V, Ezzo J, Berman B. How valuable is effort to contact authors to obtain missing data in systematic reviews. 7th Annual Cochrane Colloquium Abstracts, October 1999 in Rome. 1999.
Hetherington 1987 {published data only}
    1. Hetherington J. An international survey to identify unpublished and ongoing perinatal trials [abstract]. Controlled Clinical Trials 1987;8:287.
    1. Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989;84(2):374‐80. - PubMed
Kelly 2002 {published data only}
    1. Kelley GA, Kelley KS, Tran ZV. Retrieval of individual patient data for an exercise‐related meta‐analysis. Medicine & Science in Sports & Exercise 2002;34(5 (Suppl 1)):S225.
Kelly 2004 {published data only}
    1. Kelley GA, Kelley KS, Tran ZV. Retrieval of missing data for meta‐analysis: a practical example. International Journal of Technology Assessment in Health Care 2004;20(3):296–9. - PMC - PubMed
McGrath 1998 {published data only}
    1. McGrath J, Davies G, Soares K. Writing to authors of systematic reviews elicited further data in 17% of cases. BMJ 1998;316:631. - PMC - PubMed
Reveiz 2004 {published data only}
    1. Reveiz L, Andres Felipe C, Egdar Guillermo O. Using e‐mail for identifying unpublished and ongoing clinical trials and those published in non‐indexed journals. 12th Cochrane Colloquium: Bridging the Gaps; 2004 Oct 2‐6; Ottawa, Ontario, Canada. 2004.
    1. Reveiz L, Cardona AF, Ospina EG, Agular S. An e‐mail survey identified unpublished studies for systematic reviews. Journal of Clinical Epidemiology 2006;59(7):755‐8. - PubMed
Wille‐Jorgensen 2001 {published data only}
    1. Wille‐Jorgensen. Problems with retrieving original data: is it a selection bias?. 9th Annual Cochrane Colloquium, Lyon. October 2001.

Additional references

Chapman 2010
    1. Chapman S, Eisinga A, Clarke MJ, Hopewell S. Passport to publication? Do methodologists publish after Cochrane Colloquia?. Joint Cochrane and Campbell Colloquium. 2010 Oct 18‐22; Keystone, Colorado, USA. Cochrane Database of Systematic Reviews, Supplement 2010; Suppl: 14.
Greenhalgh 2005
    1. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 205;331(7524):1064‐5. - PMC - PubMed
Higgins 2009
    1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.2 [updated September 2009]. The Cochrane Collaboration, 2009. Available from www.cochrane‐handbook.org. The Cochrane Collaboration, 2008. Available from www.cochrane‐handbook.org.
Hopewell 2007
    1. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta‐analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews 2007, Issue 2. [DOI: 10.1002/14651858.MR000010.pub3] - DOI - PMC - PubMed
Horsley 2011
    1. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews 2011, Issue 8. [DOI: 10.1002/14651858.MR000026.pub2] - DOI - PMC - PubMed
Kirkham 2010
    1. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smith R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365. - PubMed
Lefebvre 2008
    1. Lefebvre C, Manheimer E, Glanville J on behalf of the Cochrane Information Retrieval Methods Group. Chapter 6: Searching for studies. In: Higgins JPT, Green S (eds). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.0 [updated February 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane‐handbook.org.
Scherer 2007
    1. Scherer RW, Langenberg P, Elm E. Full publication of results initially presented in abstracts. Cochrane Database of Systematic Reviews 2007, Issue 2. [DOI: 10.1002/14651858.MR000005.pub3] - DOI - PubMed
Smith 1998
    1. Smith GD, Egger M. Meta‐analysis. Unresolved issues and future developments. BMJ 1998;316(7126):221‐5. - PMC - PubMed
Song 2000
    1. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technology Assessment 2000;4(10):1‐115. - PubMed
Sterne 2008
    1. Sterne JAC, Egger M, Moher D (editors). Chapter 10: Addressing reporting biases. In: Higgins JPT, Green S (editors). Cochrane Handbook of Systematic Reviews of Interventions. Version 5.0.0 [updated February 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane‐handbook.org.

Publication types

LinkOut - more resources