Semantic Clinical Artificial Intelligence (SCAI) Usability Testing

Abstract

We evaluated the performance of Semantic Clinical Artificial Intelligence (SCAI, pronounced Sky), a large language model (LLM) knowledge resource through usability testing. This pretest-intervention-posttest mixed-methods user interface (UI) design study investigates usability to determine whether the LLM provides a more comprehensive, efficient, and enhanced user-friendly means of delivering end user information as compared to using primary sources of information from the Internet (Web). Our analysis focused on assessing the LLM's efficiency and usability in helping users arrive at accurate and reliable outcomes, to ultimately determine its value as an innovative tool for packaging and presenting information. Usability test sessions were conducted using the cognitive walkthrough approach, via Zoom. Participants were asked to respond to scenarios using only the LLM, and then only the web, and vice versa. These sessions were followed by user feedback sessions where participants rated their experiences and responded to open-ended questions related to the overall usability and satisfaction with SCAI.

Keywords: human-centered design; large language models; usability testing.