Developing a standardized framework for evaluating health apps using natural language processing

Abstract

Despite regulatory efforts, many smartphone health applications remain unregulated, raising concerns about privacy, security, and evidence-based effectiveness. The lack of standardized regulation has led to the proliferation of over 130 frameworks, introducing new criteria and methodologies for app evaluation. The sheer number of frameworks, coupled with their varying approaches to app evaluation, create challenges for comparison. Our study aims to synthesize existing knowledge and propose a standardized app evaluation framework. We conducted a synthesis of reviews on health app evaluation frameworks. Using natural language processing (NLP), we analyzed evaluation domains and grouped them into clusters based on semantic similarities. Standardized definitions for these clusters were developed. We identified eight review articles that met the inclusion criteria, each proposing between six and 17 app evaluation domains. Using NLP, we identified five clusters of app evaluation: Effectiveness & Development, Technology & Functionality, Validity & Legal, Safety & Privacy, and Implementation & Ethics, each of which was assigned a standardized definition. The clusters align with but expand on the American Psychiatric Association's evaluation domains, incorporating critical aspects such as inclusivity, safety, engagement, and ethical principles. Temporal analysis revealed an increasing focus on Effectiveness & Development, while Safety & Privacy showed a stagnation in attention over time.

Keywords: App evaluation framework; Information management; Mobile health apps; Natural Language processing; Telemedicine.

Fig. 1

In the presented figure, app evaluation criteria are hierarchically grouped into domains and broader clusters to enable a standardized evaluation. The app evaluation criteria used by individual frameworks are synthesized into app evaluation domains, as presented in various reviews. These domains are further synthesized into app evaluation clusters through our NLP-based approach.

Fig. 2

The prisma diagram presents the review articles on mHealth app evaluation frameworks included in this review.

Fig. 3

The figure visualizes temporal trends in standardized mHealth app evaluation cluster occurrences (2008–2022). The Effectiveness & Development cluster experienced an upward trend over time. Technology & Functionality and Validity & Legal declined in popularity.The mean number of reviews per year included was 4.33 (SD = 1.70), with a minimum of 2 and a maximum of 7 reviews per year.

Fig. 4

The figure compares the identified clusters with the app evaluation framework of the American Psychiatric Association (APA).

Fig. 5

The figure visualizes the process of clustering and defining the mHealth app evaluation domains used in the included reviews. The top rows shows the theoretical process, the row below presents an example.