Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Apr 16;22(4):e15474.
doi: 10.2196/15474.

Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

Elena Koneska  1 , Duncan Appelbe  1   2 , Paula R Williamson  1 , Susanna Dodd  1
Affiliations

Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

Elena Koneska et al. J Med Internet Res. .

Abstract

Background: The evaluation of web-based interventions (defined as an intervention that can be downloaded or accessed on the internet through a web browser) in randomized controlled trials (RCTs) has increased over the past two decades. Little is known about how participants' use of the intervention is measured, reported, and analyzed in these studies.

Objective: This study aimed to review the evaluation of web-based interventions in RCTs, assessing study characteristics and the methods used to record, and adjust for, intervention usage.

Methods: A systematic review of the literature was undertaken to identify all published reports of RCTs that involved a web-based intervention. A random sample of 100 published trials was selected for detailed data extraction. Information on trial characteristics was extracted, including whether web usage data were recorded, and if so, the methods used to gather these data and whether these data were used to inform efficacy analyses.

Results: A PubMed search identified 812 trials of web-based interventions published up to the end of 2017 and demonstrated a growing trend over time. Of the 100 studies reviewed, 90 studies collected web usage data, but more than half (49/90, 54%) of these studies did not state the method used for recording web usage. Only four studies attempted to check on the reliability of their web usage data collection methods. A total of 39% (35/90) studies reported patterns or levels of web intervention use, of which 21% (19/90) studies adjusted for intervention use in their outcome analysis, but only two of these used appropriate statistical methods.

Conclusions: Trialists frequently report a measure of web-based intervention usage but do not always report the collection method or provide enough detail on their analysis of web usage. Appropriate statistical methods to account for intervention use are rarely used and are not well reported even in the very few trials in which they are used. The number of trialists who attempt to check on the reliability of their web usage collection methods is extremely low.

Keywords: internet; randomized controlled trial; systematic review; web usage data; web-based interventions.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Search and screening process. RCT: randomized controlled trial.
Figure 2
Figure 2
Number of published and sampled trials of online intervention trials each year.
See this image and copyright information in PMC

References

    1. Paul J, Seib R, Prescott T. The internet and clinical trials: background, online resources, examples and issues. J Med Internet Res. 2005 Mar 16;7(1):e5. doi: 10.2196/jmir.7.1.e5. https://www.jmir.org/2005/1/e5/ - DOI - PMC - PubMed
    1. Stanley K. Design of randomized controlled trials. Circulation. 2007;115(9):1164–9. doi: 10.1161/circulationaha.105.594945. - DOI - PubMed
    1. Kendall JM. Designing a research project: randomised controlled trials and their principles. Emerg Med J. 2003 Mar;20(2):164–8. doi: 10.1136/emj.20.2.164. http://emj.bmj.com/cgi/pmidlookup?view=long&pmid=12642531 - DOI - PMC - PubMed
    1. Kabisch M, Ruckes C, Seibert-Grafe M, Blettner M. Randomized controlled trials: part 17 of a series on evaluation of scientific publications. Dtsch Arztebl Int. 2011 Sep;108(39):663–8. doi: 10.3238/arztebl.2011.0663. doi: 10.3238/arztebl.2011.0663. - DOI - DOI - PMC - PubMed
    1. Mathieu E, McGeechan K, Barratt A, Herbert R. Internet-based randomized controlled trials: a systematic review. J Am Med Inform Assoc. 2013 May 1;20(3):568–76. doi: 10.1136/amiajnl-2012-001175. http://europepmc.org/abstract/MED/23065196 - DOI - PMC - PubMed

Publication types

MeSH terms