Key Variables for Effective eHealth Designs for Individuals With and Without Mental Health Disorders: 2^12-4 Fractional Factorial Experiment

Abstract

Background: eHealth applications not only offer the potential to increase service convenience and responsiveness but also expand the ability to tailor services to improve relevance, engagement, and use. To achieve these goals, it is critical that the designs are intuitive. Limited research exists on designs that work for those with a severe mental illness (SMI), many of whom have difficulty traveling for treatments, reject or infrequently seek treatment, and tend to discontinue treatments for significant periods.

Objective: This study aims to evaluate the influence of 12 design variables (eg, navigational depth, reading level, and use of navigational lists) on the usability of eHealth application websites for those with and without SMI.

Methods: A 2^12-4 fractional factorial experiment was used to specify the designs of 256 eHealth websites. This approach systematically varied the 12 design variables. The final destination contents of all websites were identical, and only the designs of the navigational pages varied. The 12 design elements were manipulated systematically to allow the assessment of combinations of design elements rather than only one element at a time. Of the 256 websites, participants (n=222) sought the same information on 8 randomly selected websites. Mixed effect regressions, which accounted for the dependency of the 8 observations within participants, were used to test for main effects and interactions on the ability and time to find information. Classification and regression tree analyses were used to identify effects among the 12 variables on participants' abilities to locate information, for the sample overall and each of the 3 diagnostic groups of participants (schizophrenia spectrum disorder [SSD], other mental illnesses, and no mental illness).

Results: The best and worst designs were identified for each of these 4 groups. The depth of a website's navigation, that is, the number of screens users needed to navigate to find the desired content, had the greatest influence on usability (ability to find information) and efficiency (time to find information). The worst performing designs for those with SSD had a 9% success rate, and the best had a 51% success rate: the navigational designs made a 42% difference in usability. For the group with other mental illnesses, the design made a 50% difference, and for those with no mental illness, a 55% difference was observed. The designs with the highest usability had several key design similarities, as did those with the poorest usability.

Conclusions: It is possible to identify evidence-based strategies for designing eHealth applications that result in significantly better performance. These improvements in design benefit all users. For those with SSD or other SMIs, there are designs that are highly effective. Both the best and worst designs have key similarities but vary in some characteristics.

Keywords: eHealth; eHealth design; fractional factorial design; schizophrenia; severe mental illness; usability; website; website design; website usability.

Figure 1

Consolidated Standards of Reporting Trials chart. WRAT: Wide-Range Achievement Test.

Figure 2

Classification and regression trees analyses of all participants.

Figure 3

Classification and regression trees analyses participants with schizophrenia spectrum disorder.

Figure 4

Classification and regression trees analyses of participants with a mental illness other than schizophrenia spectrum disorder.

Figure 5

Classification and regression trees analyses of participants with no mental illness.