A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity

Abstract

Objectives: Identifying pragmatic trials from among all randomized trials is challenging because of inconsistent reporting. Our objective was to develop and validate a search filter to identify reports of pragmatic trials from Ovid MEDLINE.

Study design and setting: Two sets of known and probable pragmatic trial records were analyzed using text mining to generate candidate terms. Two large population sets comprising clinical trials and explanatory trials were used to select discriminating terms. Various combinations of terms were tested iteratively to achieve optimal search performance. Two externally derived sets were used to validate sensitivity and specificity of the derived filters.

Results: Our validated sensitivity-maximizing filter (combines trial design terms with terms relating to attributes of pragmatic trials) retrieves over 42,000 records in MEDLINE and has sensitivity of 46.4% (95% confidence interval (CI) 37.2 to 55.7%) and estimated specificity of 98.1% (95% CI 93.4 to 99.8%). Search performance is superior to other ad hoc filters for pragmatic trials. The Cochrane search for randomized trials has much better sensitivity (98.2%), but poorer specificity (1.9%) and retrieves 4.5 million records.

Conclusion: A highly specific filter (low false positive rate) with moderate sensitivity is available for identifying reports of trials more likely to be pragmatic.

Keywords: Database searching; Explanatory trials; MEDLINE; Pragmatic trials; Search filters; Search strategies; Specificity.