Development of a Flexible Chain of Thought Framework for Automated Routing of Patient Portal Messages

Abstract

The increase in utilization of patient portal messages has imposed a considerable burden on healthcare providers, contributing to an increased incidence of provider burnout. This study introduces a framework for leveraging Large Language Models (LLMs) and Chain-of-Thought (CoT) prompting in order to automatically categorize and route messages to their appropriate location. The modeling framework, which utilizes gold standard annotations from triage nurses, not only facilitates the dynamic adaptation of the model to evolving healthcare workflows and emerging edge-case scenarios, but also significantly improves the model's classification accuracy compared to traditional zero-shot methods. In addition, the framework allows for flexibility in its task and continuous improvement via annotation of exemplar messages. The model is able to accurately categorize messages in an automated fashion, which has potential to dramatically ease the burden on providers and provide faster and safer responses to patients. This framework can also be readily extended to work in a variety of clinical and documentation settings.

Figure 1:

An example gold-standard message, which includes the patient message, the reasoning for why the message was routed to its eventual location, and the answer.

Figure 2:

Performance by category for KNN + CoT + Generated Response. Correctly classified counts are located on the diagonals. Messages which were predicted as triage are not considered as misclassified, as they represent the status quo.

Figure 3:

Improving model performance with exemplar messages. Exemplar messages in the candidate set for CoT retrieval allow for continuous improvement of the solution, since previously misclassified messages are subsequently correctly classified.