Boolean versus ranked querying for biomedical systematic reviews

Abstract

Background: The process of constructing a systematic review, a document that compiles the published evidence pertaining to a specified medical topic, is intensely time-consuming, often taking a team of researchers over a year, with the identification of relevant published research comprising a substantial portion of the effort. The standard paradigm for this information-seeking task is to use Boolean search; however, this leaves the user(s) the requirement of examining every returned result. Further, our experience is that effective Boolean queries for this specific task are extremely difficult to formulate and typically require multiple iterations of refinement before being finalized.

Methods: We explore the effectiveness of using ranked retrieval as compared to Boolean querying for the purpose of constructing a systematic review. We conduct a series of experiments involving ranked retrieval, using queries defined methodologically, in an effort to understand the practicalities of incorporating ranked retrieval into the systematic search task.

Results: Our results show that ranked retrieval by itself is not viable for this search task requiring high recall. However, we describe a refinement of the standard Boolean search process and show that ranking within a Boolean result set can improve the overall search performance by providing early indication of the quality of the results, thereby speeding up the iterative query-refinement process.

Conclusions: Outcomes of experiments suggest that an interactive query-development process using a hybrid ranked and Boolean retrieval system has the potential for significant time-savings over the current search process in the systematic reviewing.

Figure 1

A sample ranked query (TRC) based on an AHRQ systematic review (Drug Dataset). Note the headings only show where the query words are sourced and do not represent any structure to the query. The query is treated as a "bag-of-words", that is, the entire text is entered as a query without imposing any structure or word order.

Figure 2

Tolerance versus recall (left), and rank (right). Lines show the relationship for the best (query 14), worst (query 1), and average over 15 TRC queries, when retrieving from the full MEDLINE collection. Evaluations are based on final included studies (Tier 2).

Figure 3

Average recall versus precision using TRC ranked queries on MEDLINE.

Figure 4

Recall of Boolean queries after re-running on Ovid MEDLINE (1950-Nov week 3, 2008). Average recall for (a) original queries (upper) are 0.27 and 0.29 for Tier 1 and Tier 2, respectively, and for (b) simplified queries (lower) are 0.79 and 0.82.

Figure 5

Tolerance versus recall (left), and rank (right) for two queries (best and worst), and average over 15 TRC queries. Retrieval was over subsets of MEDLINE determined by the simplified Boolean queries. Evaluations are based on final included studies (Tier 2).

Figure 6

Average recall versus precision using TRC ranked queries on the subset of MEDLINE retrieved by their corresponding Boolean queries.

Figure 7

Recall of the Boolean queries in the Misc dataset after running the original (upper) and simplified (lower) Boolean queries on OVID MEDLINE.