Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Mar 13;22(1):67.
doi: 10.1186/s12874-022-01556-2.

Using a consistency check during data collection to identify invalid responding in an online cannabis screening survey

Christina Schell  1   2 , Alexandra Godinho  1 , John A Cunningham  3   4   5
Affiliations

Using a consistency check during data collection to identify invalid responding in an online cannabis screening survey

Christina Schell et al. BMC Med Res Methodol. .

Abstract

Background: Inconsistent responding is a type of invalid responding, which occurs on self-report surveys and threatens the reliability and validity of study results. This secondary analysis evaluated the utility of identifying inconsistent responses as a real-time, direct method to improve quality during data collection for an Internet-based RCT.

Methods: The cannabis subscale of the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) was administered as part of eligibility screening for the RCT. Following the consent procedure, the cannabis subscale was repeated during the baseline interview. Responses were automatically compared and individuals with inconsistent responses were screened out.

Results: Nearly half of those initially eligible for the RCT were subsequently screened out for data quality issues (n = 626, 45.3%). Between-group bivariate analysis found that those screened out (OUT) were significantly older (OUT = 39.5 years (SD = 13.9), IN = 35.7 years (SD = 12.9), p < .001), more had annual incomes less than $20,000CND (OUT = 58.3%, IN = 53.0%, p = .047), used cannabis less often in the past 30 days (OUT = 23.3 days (SD = 9.7), IN = 24.8 days (SD = 11.3), p < .006), and had lower total ASSIST scores at screener (OUT = 19.3 (SD = 8.0), IN = 23.8 (SD = 10.4), p < .001) and baseline (OUT = 17.5 (SD = 7.9), IN = 23.3 (SD = 10.3), p < .001) compared to participants who were screened in to the RCT.

Conclusion: Inconsistent responding may occur at high rates in Internet research and direct methods to identify invalid responses are needed. Comparing responses for consistency can be programmed in Internet surveys to automatically screen participants during recruitment and reduce the need for post-hoc data cleaning.

Keywords: Cannabis; Inconsistency; Invalid responding.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no conflicts of interest to report.

Figures

Fig. 1
Fig. 1
Consort chart
See this image and copyright information in PMC

References

    1. Meade A, Craig S. Identifying careless responses in survey data. Psychol Methods. 2012;17(3):437–455. doi: 10.1037/a0028085. - DOI - PubMed
    1. Huang J, Curran P, Keeney J, Poposki E, DeShon R. Detecting and deterring insufficient effort responding to surveys. J Bus Psychol. 2012;27:99–114. doi: 10.1007/s10869-011-9231-8. - DOI
    1. Godinho A, Kushnir V, Cunningham J. Unfaithful findings: Identifying careless responding in addictions research. Addiction. 2016;111:955–6. doi: 10.1111/add.13221. - DOI - PubMed
    1. Johnson J. Ascertaining the validity of individual protocols from web-based personality inventories. J Res Pers. 2005;39:103–129. doi: 10.1016/j.jrp.2004.09.009. - DOI
    1. Conijn J, Franz G, Emons W, deBeurs E, Carlier I. The assessment and impact of careless responding in routine outcome monitoring within mental health care. Multivar Behav Res. 2019;64(4):593–611. doi: 10.1080/00273171.2018.1563520. - DOI - PubMed

Publication types

LinkOut - more resources