Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Jan 6;24(1):e28368.
doi: 10.2196/28368.

A New Remote Guided Method for Supervised Web-Based Cognitive Testing to Ensure High-Quality Data: Development and Usability Study

Victoria Leong  1   2   3 , Kausar Raheel  1 , Jia Yi Sim  1 , Kriti Kacker  1 , Vasilis M Karlaftis  4 , Chrysoula Vassiliu  5 , Kastoori Kalaivanan  2 , S H Annabel Chen  1   2   3   6 , Trevor W Robbins  4   7 , Barbara J Sahakian  7   8 , Zoe Kourtzi  4
Affiliations

A New Remote Guided Method for Supervised Web-Based Cognitive Testing to Ensure High-Quality Data: Development and Usability Study

Victoria Leong et al. J Med Internet Res. .

Abstract

Background: The global COVID-19 pandemic has triggered a fundamental reexamination of how human psychological research can be conducted safely and robustly in a new era of digital working and physical distancing. Online web-based testing has risen to the forefront as a promising solution for the rapid mass collection of cognitive data without requiring human contact. However, a long-standing debate exists over the data quality and validity of web-based studies. This study examines the opportunities and challenges afforded by the societal shift toward web-based testing and highlights an urgent need to establish a standard data quality assurance framework for online studies.

Objective: This study aims to develop and validate a new supervised online testing methodology, remote guided testing (RGT).

Methods: A total of 85 healthy young adults were tested on 10 cognitive tasks assessing executive functioning (flexibility, memory, and inhibition) and learning. Tasks were administered either face-to-face in the laboratory (n=41) or online using remote guided testing (n=44) and delivered using identical web-based platforms (Cambridge Neuropsychological Test Automated Battery, Inquisit, and i-ABC). Data quality was assessed using detailed trial-level measures (missed trials, outlying and excluded responses, and response times) and overall task performance measures.

Results: The results indicated that, across all data quality and performance measures, RGT data was statistically-equivalent to in-person data collected in the lab (P>.40 for all comparisons). Moreover, RGT participants out-performed the lab group on measured verbal intelligence (P<.001), which could reflect test environment differences, including possible effects of mask-wearing on communication.

Conclusions: These data suggest that the RGT methodology could help ameliorate concerns regarding online data quality-particularly for studies involving high-risk or rare cohorts-and offer an alternative for collecting high-quality human cognitive data without requiring in-person physical attendance.

Keywords: COVID-19; executive functions; learning; neurocognitive assessment; web-based testing.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: TWR receives consultancy fees from Cambridge Cognition. BJS consults for Cambridge Cognition.

Figures

Figure 1
Figure 1
Overview of remote guided and face-to-face testing processes.
Figure 2
Figure 2
Hardware specifications for remote guided participants (total N=44), including computer (a) brand; (b) operating system; (c) screen size (in inches) (d) screen resolution (in pixels); (e) processor and (f) RAM (in GB).
Figure 3
Figure 3
Web capability for remote guided participants (total n=44), including (a) internet download/upload speed (higher=better); and (b) internet latency (shorter=better).
Figure 4
Figure 4
Plot of performance indices for (a) i-ABC; (b) Inquisit; (c) CANTAB and (d) Verbally delivered tasks. Face-to-face participants are shown in dark grey bars, remote guided participants are shown in light grey bars. Error bars indicate the standard error of the mean, ***P<.001.
Figure 5
Figure 5
Summary of considerations for suitability of unsupervised, supervised web testing and in-person methodologies for cognitive testing. RT: reaction time.
See this image and copyright information in PMC

References

    1. Omary M, Eswaraka J, Kimball SD, Moghe PV, Panettieri RA, Scotto KW. The COVID-19 pandemic and research shutdown: staying safe and productive. J Clin Invest. 2020 Jun 01;130(6):2745–2748. doi: 10.1172/JCI138646. doi: 10.1172/JCI138646.138646 - DOI - DOI - PMC - PubMed
    1. Ali SH, Foreman J, Capasso A, Jones AM, Tozan Y, DiClemente RJ. Social media as a recruitment platform for a nationwide online survey of COVID-19 knowledge, beliefs, and practices in the United States: methodology and feasibility analysis. BMC Med Res Methodol. 2020 May 13;20(1):116. doi: 10.1186/s12874-020-01011-0. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-... 10.1186/s12874-020-01011-0 - DOI - DOI - PMC - PubMed
    1. Serhan D. Transitioning from face-to-face to remote learning: Students attitudes and perceptions of using Zoom during COVID-19 pandemic. International Journal of Technology in Education and Science. 2020:335–342. doi: 10.46328/ijtes.v4i4.148. - DOI
    1. Ray S, Srivastava S. Virtualization of science education: a lesson from the COVID-19 pandemic. J Proteins Proteom. 2020 May 31;11(2):1–4. doi: 10.1007/s42485-020-00038-7. http://europepmc.org/abstract/MED/33132627 38 - DOI - PMC - PubMed
    1. Cinelli M, Quattrociocchi W, Galeazzi A, Valensise CM, Brugnoli E, Schmidt AL, Zola P, Zollo F, Scala A. The COVID-19 social media infodemic. Sci Rep. 2020 Oct 06;10(1):16598. doi: 10.1038/s41598-020-73510-5. doi: 10.1038/s41598-020-73510-5.10.1038/s41598-020-73510-5 - DOI - DOI - PMC - PubMed

Publication types