Identifying relevant studies for systematic reviews

Abstract

Objective: To examine the sensitivity and precision of Medline searching for randomised clinical trials.

Design: Comparison of results of Medline searches to a "gold standard" of known randomised clinical trials in ophthalmology published in 1988; systematic review (meta-analysis) of results of similar, but separate, studies from many fields of medicine.

Populations: Randomised clinical trials published in 1988 in journals indexed in Medline, and those not indexed in Medline and identified by hand search, comprised the gold standard. Gold standards for the other studies combined in the meta-analysis were based on: randomised clinical trials published in any journal, whether indexed in Medline or not; those published in any journal indexed in Medline; or those published in a selected group of journals indexed in Medline.

Main outcome measure: Sensitivity (proportion of the total number of known randomised clinical trials identified by the search) and precision (proportion of publications retrieved by Medline that were actually randomised clinical trials) were calculated for each study and combined to obtain weighted means. Searches producing the "best" sensitivity were used for sensitivity and precision estimates when multiple searches were performed.

Results: The sensitivity of searching for ophthalmology randomised clinical trials published in 1988 was 82%, when the gold standard was for any journal, 87% for any journal indexed in Medline, and 88% for selected journals indexed in Medline. Weighted means for sensitivity across all studies were 51%, 77%, and 63%, respectively. The weighted mean for precision was 8% (median 32.5%). Most searchers seemed not to use freetext subject terms and truncation of those terms.

Conclusion: Although the indexing terms available for searching Medline for randomised clinical trials have improved, sensitivity still remains unsatisfactory. A mechanism is needed to "'register" known trials, preferably by retrospective tagging of Medline entries, and incorporating trials published before 1966 and in journals not indexed by Medline into the system.