Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 Feb 10;4(1):e15.
doi: 10.2196/mhealth.5176.

Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps

Adam C Powell  1 , John Torous, Steven Chan, Geoffrey Stephen Raynor, Erik Shwarts, Meghan Shanahan, Adam B Landman
Affiliations

Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps

Adam C Powell et al. JMIR Mhealth Uhealth. .

Abstract

Background: There are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves.

Objective: We sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability.

Methods: We identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff's alpha was calculated for each of the measures and reported by app category and in aggregate.

Results: The measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps.

Conclusions: We found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.

Keywords: evaluation studies; health apps; mental health; mobile applications; ratings.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: ACP has an ownership stake in ArxViva, Inc. and Payer+Provider Syndicate and is on the scientific advisory board of PsyberGuide. SC is an Associate Editor of iMedicalApps. ABL has support from the Commonwealth Fund and is an advisor to the Hacking Medicine Institute.

References

    1. IMS Health. 2015. [2015-09-24]. IMS Institute for Healthcare Economics http://www.imshealth.com/portal/site/imshealth/menuitem.0be132395225d98e... .
    1. Powell AC, Landman AB, Bates DW. In search of a few good apps. JAMA. 2014 May 14;311(18):1851–1852. doi: 10.1001/jama.2014.2564.1852662 - DOI - PubMed
    1. Powell AC, Landman AB, Bates DW. Certification of mobile apps for health care--reply. JAMA. 2014 Sep 17;312(11):1156–1157. doi: 10.1001/jama.2014.9007.1904820 - DOI - PubMed
    1. Shen N, Levitan M, Johnson A, Bender JL, Hamilton-Page M, Jadad Alejandro AR, Wiljer D. Finding a depression app: a review and content analysis of the depression app marketplace. JMIR Mhealth Uhealth. 2015;3(1):e16. doi: 10.2196/mhealth.3713. http://mhealth.jmir.org/2015/1/e16/ v3i1e16 - DOI - PMC - PubMed
    1. Torous J, Powell AC. Current research and trends in the use of smartphone applications for mood disorders. Internet Interventions. 2015 May;2(2):169–173. doi: 10.1016/j.invent.2015.03.002. - DOI

LinkOut - more resources