Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Feb:135:105039.
doi: 10.1016/j.envint.2019.105039. Epub 2019 Dec 18.

RoB-SPEO: A tool for assessing risk of bias in studies estimating the prevalence of exposure to occupational risk factors from the WHO/ILO Joint Estimates of the Work-related Burden of Disease and Injury

Frank Pega  1 , Susan L Norris  2 , Claudine Backes  3 , Lisa A Bero  4 , Alexis Descatha  5 , Diana Gagliardi  6 , Lode Godderis  7 , Tom Loney  8 , Alberto Modenese  9 , Rebecca L Morgan  10 , Daniela Pachito  11 , Marilia B S Paulo  12 , Paul T J Scheepers  13 , Vivi Schlünssen  14 , Daria Sgargi  15 , Ellen K Silbergeld  16 , Kathrine Sørensen  17 , Patrice Sutton  18 , Thomas Tenkate  19 , Denise Torreão Corrêa da Silva  20 , Yuka Ujita  21 , Emilie van Deventer  3 , Tracey J Woodruff  18 , Daniele Mandrioli  15
Affiliations

RoB-SPEO: A tool for assessing risk of bias in studies estimating the prevalence of exposure to occupational risk factors from the WHO/ILO Joint Estimates of the Work-related Burden of Disease and Injury

Frank Pega et al. Environ Int. 2020 Feb.

Abstract

Background: The World Health Organization (WHO) and the International Labour Organization (ILO) are developing joint estimates of the work-related burden of disease and injury (WHO/ILO Joint Estimates). For this, systematic reviews of studies estimating the prevalence of exposure to selected occupational risk factors will be conducted to provide input data for estimations of the number of exposed workers. A critical part of systematic review methods is to assess risk of bias (RoB) of individual studies. In this article, we present and describe the development of such a tool, called the Risk of Bias in Studies estimating Prevalence of Exposure to Occupational risk factors (RoB-SPEO) tool; report results from RoB-SPEO's pilot testing; note RoB-SPEO's limitations; and suggest how the tool might be tested and developed further.

Methods: Selected existing RoB tools used in environmental and occupational health systematic reviews were reviewed and analysed. From existing tools, we identified domains for the new tool and, if necessary, added new domains. For each domain, we then identified and integrated components from the existing tools (i.e. instructions, domains, guiding questions, considerations, ratings and rating criteria), and, if necessary, we developed new components. Finally, we elicited feedback from other systematic review methodologists and exposure scientists and agreed upon RoB-SPEO. Nine experts pilot tested RoB-SPEO, and we calculated a raw measure of inter-rater agreement (Pi) for each of its domain, rating Pi < 0.4 as poor, 0.4 ≤ Pi ≥ 0.8 as substantial and Pi > 0.80 as almost perfect agreement.

Results: Our review found no standard tool for assessing RoB in prevalence studies of exposure to occupational risk factors. We identified six existing tools for environmental and occupational health systematic reviews and found that their components for assessing RoB differ considerably. With the new RoB-SPEO tool, assessors judge RoB for each of eight domains: (1) bias in selection of participants into the study; (2) bias due to a lack of blinding of study personnel; (3) bias due to exposure misclassification; (4) bias due to incomplete exposure data; (5) bias due to conflict of interest; (6) bias due to selective reporting of exposures; (7) bias due to difference in numerator and denominator; and (8) other bias. The RoB-SPEO's ratings are low, probably low, probably high, high or no information. Pilot testing of the RoB-SPEO tool found substantial inter-rater agreement for six domains (range of Pi for these domains: 0.51-0.80), but poor agreement for two domains (i.e. Pi of 0.31 and 0.33 for biases due to incomplete exposure data and in selection of participants into the study, respectively). Limitations of RoB-SPEO include that it has not yet been fully performance-tested.

Conclusions: We developed the RoB-SPEO tool for assessing RoB in prevalence studies of exposure to occupational risk factors. The tool will be applied and its performance tested in the ongoing systematic reviews for the WHO/ILO Joint Estimates.

PubMed Disclaimer

Conflict of interest statement

Susan L. Norris is a member of the GRADE working group.

Figures

Fig. 1
Fig. 1
Steps of the systematic review process.
Fig. 2
Fig. 2
Development of the RoB-SPEO tool.
Fig. 3
Fig. 3
Instructions (in Italics) and table for recording the assessment.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Armijo-Olivo S., Stiles C.R., Hagen N.A., Biondo P.D., Cummings G.G. Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J. Eval. Clin. Pract. 2012;18(1):12–18. - PubMed
    1. Bero L. What is in a name? Nonfinancial influences on the outcomes of systematic reviews and guidelines. J. Clin. Epidemiol. 2014;67(11):1239–1241. - PubMed
    1. Bero L. Addressing bias and conflict of interest among biomedical researchers. JAMA. 2017;317(17):1723–1724. - PubMed
    1. Bero L.A., Grundy Q. Why having a (nonfinancial) interest is not a conflict of interest. PLoS Biol. 2016;14(12) - PMC - PubMed
    1. Bilandzic A., Fitzpatrick T., Rosella L., Henry D. Risk of bias in systematic reviews of non-randomized studies of adverse cardiovascular effects of thiazolidinediones and cyclooxygenase-2 inhibitors: application of a new Cochrane risk of bias tool. PLoS Med. 2016;13(4) - PMC - PubMed

Publication types