Evaluation of the EsteR Toolkit for COVID-19 Decision Support: Sensitivity Analysis and Usability Study

Abstract

Background: During the COVID-19 pandemic, local health authorities were responsible for managing and reporting current cases in Germany. Since March 2020, employees had to contain the spread of COVID-19 by monitoring and contacting infected persons as well as tracing their contacts. In the EsteR project, we implemented existing and newly developed statistical models as decision support tools to assist in the work of the local health authorities.

Objective: The main goal of this study was to validate the EsteR toolkit in two complementary ways: first, investigating the stability of the answers provided by our statistical tools regarding model parameters in the back end and, second, evaluating the usability and applicability of our web application in the front end by test users.

Methods: For model stability assessment, a sensitivity analysis was carried out for all 5 developed statistical models. The default parameters of our models as well as the test ranges of the model parameters were based on a previous literature review on COVID-19 properties. The obtained answers resulting from different parameters were compared using dissimilarity metrics and visualized using contour plots. In addition, the parameter ranges of general model stability were identified. For the usability evaluation of the web application, cognitive walk-throughs and focus group interviews were conducted with 6 containment scouts located at 2 different local health authorities. They were first asked to complete small tasks with the tools and then express their general impressions of the web application.

Results: The simulation results showed that some statistical models were more sensitive to changes in their parameters than others. For each of the single-person use cases, we determined an area where the respective model could be rated as stable. In contrast, the results of the group use cases highly depended on the user inputs, and thus, no area of parameters with general model stability could be identified. We have also provided a detailed simulation report of the sensitivity analysis. In the user evaluation, the cognitive walk-throughs and focus group interviews revealed that the user interface needed to be simplified and more information was necessary as guidance. In general, the testers rated the web application as helpful, especially for new employees.

Conclusions: This evaluation study allowed us to refine the EsteR toolkit. Using the sensitivity analysis, we identified suitable model parameters and analyzed how stable the statistical models were in terms of changes in their parameters. Furthermore, the front end of the web application was improved with the results of the conducted cognitive walk-throughs and focus group interviews regarding its user-friendliness.

Keywords: COVID-19; decision support tool; public health; sensitivity analysis; usability study; web application.

Figure 1

Simulation result of the infection period model: 1–intersection-over-union (IoU) of the 80% high-density region (HDR) of the infection time (A) and the distributions derived from 4 specific parameter sets (B). The default parameters are marked in red.

Figure 2

Simulation result of the infection spread model: difference in the predicted total infections after a group event. The default parameters are marked in red.

Figure 3

Simulation result of the illness period model: Wasserstein metric for the distribution of the symptom onset of the first contact generation (A) and the distributions of 3 specific parameter sets (B). The default parameters are marked in red.

Figure 4

Simulation result of the infectious period model: 1–intersection-over-union (IoU) of the 80% high-density region of the infectious period. The default parameters derived from data from the first half of 2020 are marked in red.

Figure 5

Simulation result of the risk assessment for group quarantine model for the childcare scenario (A) and the prior distributions for 2 certain parameter sets and the likelihood (B). The default parameters derived from data from the second half of 2020 are marked in red.

Figure 6

Ratings of the web application on a 5-point Likert scale, with 1 indicating full agreement with the negative attribute and 5 indicating full agreement with the positive attribute. The 6 testers are indicated in (1) purple, (2) orange, (3) turquoise, (4) blue, (5) green, and (6) gray. The median is indicated as a black square.