Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Apr;78(4):1176-1185.
doi: 10.1111/jan.15166. Epub 2022 Feb 7.

Validity and reliability of the novel three-item occupational violence patient risk assessment tool

C J Cabilan  1   2 , Joshua McRae  3 , Ben Learmont  1 , Karen Taurima  1 , Sue Galbraith  1 , Dale Mason  1 , Robert Eley  1   4 , Centaine Snoswell  5 , Amy N B Johnston  1   2
Affiliations

Validity and reliability of the novel three-item occupational violence patient risk assessment tool

C J Cabilan et al. J Adv Nurs. 2022 Apr.

Abstract

Aim: To develop and psychometrically test an occupational violence (OV) risk assessment tool in the emergency department (ED).

Design: Three studies were conducted in phases: content validity, predictive validity and inter-rater reliability from June 2019 to March 2021.

Methods: For content validity, ED end users (mainly nurses) were recruited to rate items that would appropriately assess for OV risk. Subsequently, a risk assessment tool was developed and tested for its predictive validity and inter-rater reliability. For predictive validity, triage notes of ED presentations in a month with the highest OV were assessed for presence of OV risk. Each presentation was then matched with events recorded in the OV incident register. Sensitivity and specificity values were calculated. For inter-rater reliability, two assessors-trained and untrained-independently assessed the triage notes for presence of OV risk. Cohen's kappa was calculated.

Results: Two rounds of content validity with a total of N = 81 end users led to the development of a three-domain tool that assesses for OV risk using aggression history, behavioural concerns (i.e., angry, clenched fist, demanding, threatening language or resisting care) and clinical presentation concerns (i.e., alcohol/drug intoxication and erratic cognition). Recommended risk ratings are low (score = 0 risk domain present), moderate (score = 1 risk domain present) and high (score = 2-3 risk domains present), with an area under the curve of 0.77 (95% confidence interval 0.7-0.81, p < .01). Moderate risk rating had a 61% sensitivity and 91% specificity, whereas high risk rating had 37% sensitivity and 97% specificity. Inter-rater reliability ranged from 0.67 to 0.75 (p < .01), suggesting moderate agreement.

Conclusions: The novel three-domain OV risk assessment tool was shown to be appropriate and relevant for application in EDs. The tool, developed through a rigorous content validity process, demonstrates acceptable predictive validity and inter-rater reliability.

Impact: The developed tool is currently piloted in a single hospital ED, with a view to extend to inpatient settings and other hospitals.

Keywords: emergency nursing; instrument development; psychometrics; risk assessment; risk factors; workplace violence.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

FIGURE 1
FIGURE 1
Visual overview of the study designs used to establish validity and reliability of the new occupational violence patient risk assessment tool
FIGURE 2
FIGURE 2
The Queensland Occupational Violence Patient Risk Assessment tOol (QOVPRAO, pronounced kwov‐pro)
FIGURE 3
FIGURE 3
Receiver operating characteristic (ROC) curve using of risk ratings: Low (score = 0 risk domain present), moderate (score = 1 risk domain present) and high (score = 2–3 risk domains present). The area under the curve is 0.77 (95% confidence interval 0.7–0.81, p < .01)
See this image and copyright information in PMC

References

    1. Akther, S. F. , Molyneaux, E. , Stuart, R. , Johnson, S. , Simpson, A. , & Oram, S. (2019). Patients' experiences of assessment and detention under mental health legislation: Systematic review and qualitative meta‐synthesis. BJPsych Open, 5(3), e37. 10.1192/bjo.2019.19 - DOI - PMC - PubMed
    1. Almanasreh, E. , Moles, R. , & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. 10.1016/j.sapharm.2018.03.066 - DOI - PubMed
    1. Anthoine, E. , Moret, L. , Regnault, A. , Sébille, V. , & Hardouin, J.‐B. (2014). Sample size used to validate a scale: A review of publications on newly‐developed patient reported outcomes measures. Health and Quality of Life Outcomes, 12, 176–176. 10.1186/s12955-014-0176-2 - DOI - PMC - PubMed
    1. Attia, J. (2003). Moving beyond sensitivity and specificity: Using likelihood ratios to help interpret diagnostic tests. Australian Prescriber, 26(5), 111–113. 10.18773/austprescr.2003.082 - DOI
    1. Bewick, V. , Cheek, L. , & Ball, J. (2004). Statistics review 13: Receiver operating characteristic curves. Critical Care, 8(6), 1–5. 10.1186/cc3000 - DOI - PMC - PubMed