AskHERMES: An online question answering system for complex clinical questions

Abstract

Objective: Clinical questions are often long and complex and take many forms. We have built a clinical question answering system named AskHERMES to perform robust semantic analysis on complex clinical questions and output question-focused extractive summaries as answers.

Design: This paper describes the system architecture and a preliminary evaluation of AskHERMES, which implements innovative approaches in question analysis, summarization, and answer presentation. Five types of resources were indexed in this system: MEDLINE abstracts, PubMed Central full-text articles, eMedicine documents, clinical guidelines and Wikipedia articles.

Measurement: We compared the AskHERMES system with Google (Google and Google Scholar) and UpToDate and asked physicians to score the three systems by ease of use, quality of answer, time spent, and overall performance.

Results: AskHERMES allows physicians to enter a question in a natural way with minimal query formulation and allows physicians to efficiently navigate among all the answer sentences to quickly meet their information needs. In contrast, physicians need to formulate queries to search for information in Google and UpToDate. The development of the AskHERMES system is still at an early stage, and the knowledge resource is limited compared with Google or UpToDate. Nevertheless, the evaluation results show that AskHERMES' performance is comparable to the other systems. In particular, when answering complex clinical questions, it demonstrates the potential to outperform both Google and UpToDate systems.

Conclusions: AskHERMES, available at http://www.AskHERMES.org, has the potential to help physicians practice evidence-based medicine and improve the quality of patient care.

Fig. 1

The AskHERMES architecture.

Fig. 2

An excerpt of a partial table appearing in an eMedicine article.

Fig. 3

An excerpt of a list from a guideline in National Guideline Clearinghouse.

Fig. 4

Query term-based clustering algorithm.

Fig. 5

Illustration of hierarchical clustering structure based on query terms.

Fig. 6

Sample questions that different systems perform best on.

Fig. 7

Scatter graph of quality of answer and number of words in question.

Fig. 8

AskHERMES’ answers to “What is the cause and treatment of this old man’s stomatitis?” (both focuses are covered in a succinct way).

Fig. 9

AskHERMES’ answers to the question “What is the difference between the Denver II and the regular Denver Developmental Screening Test?”.

Fig. 10

Illustration of AskHERMES’ interface for ranked answers and related questions on the query “what is the cause and treatment of this old man’s stomatitis?”.