A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care

Abstract

Background: Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining.

Objectives: This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users.

Methods: Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies.

Results: Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of "good." We reviewed persona-based case studies that illustrate functionalities of this novel prototype.

Conclusion: The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.

Fig. 1

A visual depiction of our standardized machine learning workflow. The intersecting circular shapes highlight the iterative nature of predictive model creation. The outlined badges indicate the major end-user groups that interact at each step. While the health informatician participates in all stages of the workflow, clinical and data stakeholders and the CIO participate in selecting suitable datasets for ingestion, constructing variables, and evaluating model performance over time. API, application programming interface; CIO, chief information officer; RESTful, representational state transfer.

Fig. 2

Design overview schematic. An overview of the version 2 of our proposed designs after incorporating user feedback. Each tile represents a major section of the design. The directional arrows represent our hypothesized workflow, starting from set-up of the data analysis pipeline to tracking model performance on the dashboard to launching high-performing models via the API. The labels correspond to the specific design document that provides an in-depth overview of that specific functionality. API, application programming interface.

Fig. 3

Design walkthrough, specifying data schema and definitions. Walkthrough of specifying the data schema during data ingestion and input and outcome definitions; ( A ) specifying rules for mapping CSV file names to SQL table names; ( B ) specifying rules for mapping CSV column names to SQL column names; ( C ) overview of all inputs and outcomes, prior versions tracked and accessible via menu; ( D ) SQL editor for specifying definitions with requirements list and query output preview. CSV, comma separated value.

Fig. 4

Design walkthrough, configuring models. Walkthrough of configuring models; ( A ) overview of all active models showing prior runs; ( B ) create a new model by specifying inputs, outcomes, and language environment; ( C ) dedicated code editor for each step of defining a model with ability to add additional project files and specify bash scripts to run before or after function execution

Fig. 5

Design walkthrough, tracking performance and deploying models. Walkthrough of tracking performance and deploying models; ( A ) compare aggregated performance across different outcomes, keep track of ongoing tasks; ( B ) compare model performance for a single outcome; ( C ) configure RESTful API endpoints to allow authorized applications to access trained models; ( D ) send automatically generated documentation to developers of authorized applications. API, application programming interface; RESTful, representational state transfer.

Fig. 6

Healthcare-specific design concerns. Feature designs for healthcare-specific data security and privacy concerns; ( A ) track all events across all users, configure notifications for key events; ( B ) manage access permissions, set password policies, and check server security status; ( C ) flag possible protected health information leaks for remediation; ( D ) create API access keys and manage access requests. API, application programming interface.