Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2022 Mar 11;10(3):e30691.
doi: 10.2196/30691.

Digital Contact Tracing Apps for COVID-19: Development of a Citizen-Centered Evaluation Framework

Damyanka Tsvyatkova  1 , Jim Buckley  1   2 , Sarah Beecham  1 , Muslim Chochlov  1 , Ian R O'Keeffe  1 , Abdul Razzaq  1 , Kaavya Rekanar  1 , Ita Richardson  1   2 , Thomas Welsh  1 , Cristiano Storni  1   2   3 ; COVIGILANT Group  4
Collaborators, Affiliations
Review

Digital Contact Tracing Apps for COVID-19: Development of a Citizen-Centered Evaluation Framework

Damyanka Tsvyatkova et al. JMIR Mhealth Uhealth. .

Abstract

Background: The silent transmission of COVID-19 has led to an exponential growth of fatal infections. With over 4 million deaths worldwide, the need to control and stem transmission has never been more critical. New COVID-19 vaccines offer hope. However, administration timelines, long-term protection, and effectiveness against potential variants are still unknown. In this context, contact tracing and digital contact tracing apps (CTAs) continue to offer a mechanism to help contain transmission, keep people safe, and help kickstart economies. However, CTAs must address a wide range of often conflicting concerns, which make their development/evolution complex. For example, the app must preserve citizens' privacy while gleaning their close contacts and as much epidemiological information as possible.

Objective: In this study, we derived a compare-and-contrast evaluative framework for CTAs that integrates and expands upon existing works in this domain, with a particular focus on citizen adoption; we call this framework the Citizen-Focused Compare-and-Contrast Evaluation Framework (C3EF) for CTAs.

Methods: The framework was derived using an iterative approach. First, we reviewed the literature on CTAs and mobile health app evaluations, from which we derived a preliminary set of attributes and organizing pillars. These attributes and the probing questions that we formulated were iteratively validated, augmented, and refined by applying the provisional framework against a selection of CTAs. Each framework pillar was then subjected to internal cross-team scrutiny, where domain experts cross-checked sufficiency, relevancy, specificity, and nonredundancy of the attributes, and their organization in pillars. The consolidated framework was further validated on the selected CTAs to create a finalized version of C3EF for CTAs, which we offer in this paper.

Results: The final framework presents seven pillars exploring issues related to CTA design, adoption, and use: (General) Characteristics, Usability, Data Protection, Effectiveness, Transparency, Technical Performance, and Citizen Autonomy. The pillars encompass attributes, subattributes, and a set of illustrative questions (with associated example answers) to support app design, evaluation, and evolution. An online version of the framework has been made available to developers, health authorities, and others interested in assessing CTAs.

Conclusions: Our CTA framework provides a holistic compare-and-contrast tool that supports the work of decision-makers in the development and evolution of CTAs for citizens. This framework supports reflection on design decisions to better understand and optimize the design compromises in play when evolving current CTAs for increased public adoption. We intend this framework to serve as a foundation for other researchers to build on and extend as the technology matures and new CTAs become available.

Keywords: COVID-19; contact tracing; digital contact tracing apps; digital health; evaluation; framework; health apps; mHealth; mobile health.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Phases and deliverables in the development of our Citizen-Focused Compare-and-Contrast Evaluation Framework (C3EF) for contact tracing apps (CTAs). mHealth: mobile health.
Figure 2
Figure 2
Example 1, PathCheck SafePlaces (United States) [87]: age in "Terms of Use." Example 2, Corona-Warn (Germany) [89]: understandability of interface elements. Example 3, NOVID app (United States) [88]: descriptions offered. Example 4, COVID Tracker app (Ireland) [86]: constraints for preventing errors.
Figure 3
Figure 3
Example 5, COVID Tracker app (Ireland) [86]: inconsistency of feedback when clicking on the button “Enable”.
Figure 4
Figure 4
Electoral district–level COVID-19 statistics on the Health Service Executive's COVID Tracker app (Ireland) [86].
Figure 5
Figure 5
The "pull requests" GitHub page [99] for the Health Service Executive's COVID Tracker app [86].
See this image and copyright information in PMC

References

    1. Baxter S, Goyder E, Chambers D, Johnson M, Preston L, Booth A. Interventions to improve contact tracing for tuberculosis in specific groups and in wider populations: an evidence synthesis. Health Serv Deliv Res. 2017 Jan;5(1):1–102. doi: 10.3310/hsdr05010. - DOI - PubMed
    1. Swanson KC, Altare C, Wesseh CS, Nyenswah T, Ahmed T, Eyal N, Hamblion EL, Lessler J, Peters DH, Altmann M. Contact tracing performance during the Ebola epidemic in Liberia, 2014-2015. PLoS Negl Trop Dis. 2018 Sep;12(9):e0006762. doi: 10.1371/journal.pntd.0006762. https://dx.plos.org/10.1371/journal.pntd.0006762 PNTD-D-18-00528 - DOI - DOI - PMC - PubMed
    1. Anglemyer A, Moore TH, Parker L, Chambers T, Grady A, Chiu K, Parry M, Wilczynska M, Flemyng E, Bero L. Digital contact tracing technologies in epidemics: a rapid review. Cochrane Database Syst Rev. 2020 Aug 18;8:CD013699. doi: 10.1002/14651858.CD013699. http://europepmc.org/abstract/MED/33502000 - DOI - PMC - PubMed
    1. Danquah LO, Hasham N, MacFarlane M, Conteh FE, Momoh F, Tedesco AA, Jambai A, Ross DA, Weiss HA. Use of a mobile application for Ebola contact tracing and monitoring in northern Sierra Leone: a proof-of-concept study. BMC Infect Dis. 2019 Sep 18;19(1):810. doi: 10.1186/s12879-019-4354-z. https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-019-4354-z 10.1186/s12879-019-4354-z - DOI - DOI - PMC - PubMed
    1. Ming LC, Untong N, Aliudin NA, Osili N, Kifli N, Tan CS, Goh KW, Ng PW, Al-Worafi YM, Lee KS, Goh HP. Mobile health apps on COVID-19 launched in the early days of the pandemic: content analysis and review. JMIR Mhealth Uhealth. 2020 Sep 16;8(9):e19796. doi: 10.2196/19796. https://mhealth.jmir.org/2020/9/e19796/ v8i9e19796 - DOI - PMC - PubMed

Publication types

Substances