ACCU3RATE: A mobile health application rating scale based on user reviews

Abstract

Background: Over the last decade, mobile health applications (mHealth App) have evolved exponentially to assess and support our health and well-being.

Objective: This paper presents an Artificial Intelligence (AI)-enabled mHealth app rating tool, called ACCU3RATE, which takes multidimensional measures such as user star rating, user review and features declared by the developer to generate the rating of an app. However, currently, there is very little conceptual understanding on how user reviews affect app rating from a multi-dimensional perspective. This study applies AI-based text mining technique to develop more comprehensive understanding of user feedback based on several important factors, determining the mHealth app ratings.

Method: Based on the literature, six variables were identified that influence the mHealth app rating scale. These factors are user star rating, user text review, user interface (UI) design, functionality, security and privacy, and clinical approval. Natural Language Toolkit package is used for interpreting text and to identify the App users' sentiment. Additional considerations were accessibility, protection and privacy, UI design for people living with physical disability. Moreover, the details of clinical approval, if exists, were taken from the developer's statement. Finally, we fused all the inputs using fuzzy logic to calculate the new app rating score.

Results and conclusions: ACCU3RATE concentrates on heart related Apps found in the play store and App gallery. The findings indicate the efficacy of the proposed method as opposed to the current device scale. This study has implications for both App developers and consumers who are using mHealth Apps to monitor and track their health. The performance evaluation shows that the proposed mHealth scale has shown excellent reliability as well as internal consistency of the scale, and high inter-rater reliability index. It has also been noticed that the fuzzy based rating scale, as in ACCU3RATE, matches more closely to the rating performed by experts.

Fig 1. (A) Year-wise number of mHealth apps uploaded in the google play, and (B) Smart phone penetration in the last five years, and (C) Pie chart shows various categories of apps in percentage.

Fig 2. Factors affecting mobile app review.

Fig 3. Relationship between input parameters and model parameters.

Fig 4. Flow diagram for opinion mining.

Fig 5. Word cloud for Polarity of words found in the comments of selected apps– (A) positive polarity words, and (B) negative polarity words.

Fig 6. Calculation of clinical approval value.

Fig 7. Fuzzy logic based fusion technique which combines the knowledge extracted from the users’ star rating, users’ text review, clinical approval, UI design, functionality and security & privacy and thereby generates a score.

Fig 8. Input membership and relation between the input-output in the rule base system.

Fig 9. Case study design and app selection flowchart.

Fig 10. Correlation matrix illustrating the coefficient among factors considered for the assessment of the app scale.

Fig 11. Comparison of traditional mobile app ratings and fuzzy based app ratings.

(A) Average scores with respect to each parameter for all the selected 43 apps, (B) Box plot for the average rated value for traditional, fuzzy based ratings, as well as ratings based on expert opinion, (C) app scores for traditional and fuzzy based ratings for all the selected apps, and (D) Comparison of various app scales in terms of ICC.