Evaluation of Chatbot Prototypes for Taking the Virtual Patient's History

Abstract

In medical education Virtual Patients (VP) are often applied to train students in different scenarios such as recording the patient's medical history or deciding a treatment option. Usually, such interactions are predefined by software logic and databases following strict rules. At this point, Natural Language Processing/Machine Learning (NLP/ML) algorithms could help to increase the overall flexibility, since most of the rules can derive directly from training data. This would allow a more sophisticated and individual conversation between student and VP. One type of technology that is heavily based on such algorithmic advances are chatbots or conversational agents. Therefore, a literature review is carried out to give insight into existing educational ideas with such agents. Besides, different prototypes are implemented for the scenario of taking the patient's medical history, responding with the classified intent of a generic anamnestic question. Although the small number of questions (n=109) leads to a high SD during evaluation, all scores (recall, precision, f1) reach already a level above 80% (micro-averaged). This shows a first promising step to use these prototypes for taking the medical history of a VP.

Keywords: algorithms; machine learning; medical education; natural language processing.