Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study

Abstract

Background: Increasing use of computational methods in health care provides opportunities to address previously unsolvable problems. Machine learning techniques applied to routinely collected data can enhance clinical tools and improve patient outcomes, but their effective deployment comes with significant challenges. While some tasks can be addressed by training machine learning models directly on the collected data, more complex problems require additional input in the form of data annotations. Data annotation is a complex and time-consuming problem that requires domain expertise and frequently, technical proficiency. With clinicians' time being an extremely limited resource, existing tools fail to provide an effective workflow for deployment in health care.

Objective: This paper investigates the approach of intensive care unit staff to the task of data annotation. Specifically, it aims to (1) understand how clinicians approach data annotation and (2) capture the requirements for a digital annotation tool for the health care setting.

Methods: We conducted an experimental activity involving annotation of the printed excerpts of real time-series admission data with 7 intensive care unit clinicians. Each participant annotated an identical set of admissions with the periods of weaning from mechanical ventilation during a single 45-minute workshop. Participants were observed during task completion and their actions were analyzed within Norman's Interaction Cycle model to identify the software requirements.

Results: Clinicians followed a cyclic process of investigation, annotation, data reevaluation, and label refinement. Variety of techniques were used to investigate data and create annotations. We identified 11 requirements for the digital tool across 4 domains: annotation of individual admissions (n=5), semiautomated annotation (n=3), operational constraints (n=2), and use of labels in machine learning (n=1).

Conclusions: Effective data annotation in a clinical setting relies on flexibility in analysis and label creation and workflow continuity across multiple admissions. There is a need to ensure a seamless transition between data investigation, annotation, and refinement of the labels.

Keywords: ICU; annotation software; capturing software requirements; data annotation; data labeling; intensive care; machine learning.

Figure 1

Annotating data can provide additional information and context necessary to train machine learning models.

Figure 2

The provided excerpts of time series data were formatted to resemble the interfaces of the clinical information system present at the research site (the depicted table contains data for illustrative purposes only and is trimmed for conciseness). FiO2: fraction of inspired oxygen; ICU: intensive care unit; PCWP: pulmonary capillary wedge pressure; PEEP: positive end-expiratory pressure.

Figure 3

Participants annotated data in a variety of ways using pen and paper during the workshop held in the intensive care unit.

Figure 4

Data annotation followed a cyclic process involving investigation of data, label creation, reevaluation of data with labels applied, and refinement of created annotations.