Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 May 21:3:74.
doi: 10.1038/s41746-020-0268-9. eCollection 2020.

Design and testing of a mobile health application rating tool

David M Levine  1   2 , Zoe Co  1 , Lisa P Newmark  3 , Alissa R Groisser  1 , A Jay Holmgren  4 , Jennifer S Haas  2   5 , David W Bates  1   2   3
Affiliations

Design and testing of a mobile health application rating tool

David M Levine et al. NPJ Digit Med. .

Abstract

Mobile health applications ("apps") have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps' potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with ≥4 stars and <4-stars and then rated them with THESIS to examine the tool's interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96-3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11-2.32]), interoperability (1.75 [95% CI, 1.59-1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30-1.56]). Ratings using THESIS had fair interrater reliability (κ = 0.3-0.6) and excellent scale reliability (ɑ = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.

Keywords: Diagnosis; Health policy; Health services; Therapeutics.

PubMed Disclaimer

Conflict of interest statement

Competing interestsD.M.L. is the PI of an investigator-initiated study in collaboration with the for-profit entity Biofourmis, Ltd. to refine a predictive analytics algorithm for home hospitalized patients. D.W.B. consults for EarlySense, which makes patient safety monitoring systems. He receives cash compensation from CDI (Negev), Ltd., which is a not-for-profit incubator for health IT startups. He receives equity from ValeraHealth, which makes software to help patients with chronic diseases. He receives equity from Clew which makes software to support clinical decision-making in intensive care. He receives equity from MDClone, which takes clinical data and produces deidentified versions of it. D.W.B.’s financial interests have been reviewed by Brigham and Women’s Hospital and Partners HealthCare in accordance with their institutional policies. All other authors have no disclosures.

Figures

Fig. 1
Fig. 1. App selection (all categories combined).
We selected the first four apps in each disease category. Not all apps were rated due to resource constraints. Please refer to Supplementary Figs. 1–3 for individual category selection details.
Fig. 2
Fig. 2. App ratings.
a Overall app ratings. b App ratings by category. The error bars represent 95% confidence intervals. See Supplementary Table 4 for detailed ratings.
Fig. 3
Fig. 3. Path to building and evaluating THESIS.
The methods taken in the development and evaluation of THESIS. Apps from systematic keyword search in Apple and Google stores (n = 3191).
See this image and copyright information in PMC

References

    1. The Growing Value of Digital Health. https://www.iqvia.com/institute/reports/the-growing-value-of-digital-health (2017).
    1. Demographics of Mobile Device Ownership and Adoption in the United States. Pew Research Center. http://www.pewinternet.org/fact-sheet/mobile/ (2018).
    1. Krebs P, Duncan DT. Health app use among US mobile phone owners: a national survey. JMIR mHealth uHealth. 2015;3(4):e101. doi: 10.2196/mhealth.4924. - DOI - PMC - PubMed
    1. Carroll JK, et al. Who uses mobile phone health apps and does use matter? A secondary data analytics approach. J. Med. Internet Res. 2017;19(4):e125. doi: 10.2196/jmir.5604. - DOI - PMC - PubMed
    1. Schoeppe S, et al. Efficacy of interventions that use apps to improve diet, physical activity and sedentary behaviour: a systematic review. Int. J. Behav. Nutr. Phys. Act. 2016;13(1):127. doi: 10.1186/s12966-016-0454-y. - DOI - PMC - PubMed