Usability, Engagement, and Report Usefulness of Chatbot-Based Family Health History Data Collection: Mixed Methods Analysis

Abstract

Background: Family health history (FHx) is an important predictor of a person's genetic risk but is not collected by many adults in the United States.

Objective: This study aims to test and compare the usability, engagement, and report usefulness of 2 web-based methods to collect FHx.

Methods: This mixed methods study compared FHx data collection using a flow-based chatbot (KIT; the curious interactive test) and a form-based method. KIT's design was optimized to reduce user burden. We recruited and randomized individuals from 2 crowdsourced platforms to 1 of the 2 FHx methods. All participants were asked to complete a questionnaire to assess the method's usability, the usefulness of a report summarizing their experience, user-desired chatbot enhancements, and general user experience. Engagement was studied using log data collected by the methods. We used qualitative findings from analyzing free-text comments to supplement the primary quantitative results.

Results: Participants randomized to KIT reported higher usability than those randomized to the form, with a mean System Usability Scale score of 80.2 versus 61.9 (P<.001), respectively. The engagement analysis reflected design differences in the onboarding process. KIT users spent less time entering FHx information and reported more conditions than form users (mean 5.90 vs 7.97 min; P=.04; and mean 7.8 vs 10.1 conditions; P=.04). Both KIT and form users somewhat agreed that the report was useful (Likert scale ratings of 4.08 and 4.29, respectively). Among desired enhancements, personalization was the highest-rated feature (188/205, 91.7% rated medium- to high-priority). Qualitative analyses revealed positive and negative characteristics of both KIT and the form-based method. Among respondents randomized to KIT, most indicated it was easy to use and navigate and that they could respond to and understand user prompts. Negative comments addressed KIT's personality, conversational pace, and ability to manage errors. For KIT and form respondents, qualitative results revealed common themes, including a desire for more information about conditions and a mutual appreciation for the multiple-choice button response format. Respondents also said they wanted to report health information beyond KIT's prompts (eg, personal health history) and for KIT to provide more personalized responses.

Conclusions: We showed that KIT provided a usable way to collect FHx. We also identified design considerations to improve chatbot-based FHx data collection: First, the final report summarizing the FHx collection experience should be enhanced to provide more value for patients. Second, the onboarding chatbot prompt may impact data quality and should be carefully considered. Finally, we highlighted several areas that could be improved by moving from a flow-based chatbot to a large language model implementation strategy.

Keywords: chatbots; conversational agents; crowdsourcing; digital health tools; engagement; evaluation; family health history; mixed methods; report usefulness; usability.

Figure 1

Form-based family health history tool user interface. Condition definition displayed in the red box.

Figure 2

Curious interactive test (KIT), family health history chatbot, user interface on Juji. Condition question and response from KIT with KIT Avatar in the red box.

Figure 3

Study design flow diagram consisting of stages: eligibility, consent, intervention, and assessment. CUQ: Chatbot Usability Questionnaire; FHx: family health history; KIT: curious interactive test; MTurk: Mechanical Turk; SUS: System Usability Scale.

Figure 4

Prioritization of proposed chatbot feature enhancements: gamification, media elements, and personalization.