Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Sep;128(9):95002.
doi: 10.1289/EHP6980. Epub 2020 Sep 14.

Risk of Bias Assessments and Evidence Syntheses for Observational Epidemiologic Studies of Environmental and Occupational Exposures: Strengths and Limitations

Kyle Steenland  1 , M K Schubauer-Berigan  2 , R Vermeulen  3 , R M Lunn  4 , K Straif  5   6 , S Zahm  7 , P Stewart  8 , W D Arroyave  9 , S S Mehta  4 , N Pearce  10
Affiliations

Risk of Bias Assessments and Evidence Syntheses for Observational Epidemiologic Studies of Environmental and Occupational Exposures: Strengths and Limitations

Kyle Steenland et al. Environ Health Perspect. 2020 Sep.

Abstract

Background: Increasingly, risk of bias tools are used to evaluate epidemiologic studies as part of evidence synthesis (evidence integration), often involving meta-analyses. Some of these tools consider hypothetical randomized controlled trials (RCTs) as gold standards.

Methods: We review the strengths and limitations of risk of bias assessments, in particular, for reviews of observational studies of environmental exposures, and we also comment more generally on methods of evidence synthesis.

Results: Although RCTs may provide a useful starting point to think about bias, they do not provide a gold standard for environmental studies. Observational studies should not be considered inherently biased vs. a hypothetical RCT. Rather than a checklist approach when evaluating individual studies using risk of bias tools, we call for identifying and quantifying possible biases, their direction, and their impacts on parameter estimates. As is recognized in many guidelines, evidence synthesis requires a broader approach than simply evaluating risk of bias in individual studies followed by synthesis of studies judged unbiased, or with studies given more weight if judged less biased. It should include the use of classical considerations for judging causality in human studies, as well as triangulation and integration of animal and mechanistic data.

Conclusions: Bias assessments are important in evidence synthesis, but we argue they can and should be improved to address the concerns we raise here. Simplistic, mechanical approaches to risk of bias assessments, which may particularly occur when these tools are used by nonexperts, can result in erroneous conclusions and sometimes may be used to dismiss important evidence. Evidence synthesis requires a broad approach that goes beyond assessing bias in individual human studies and then including a narrow range of human studies judged to be unbiased in evidence synthesis. https://doi.org/10.1289/EHP6980.

PubMed Disclaimer

Figures

Figure 1 is a flow diagram, having five stages. Stage 1: Problem formulation leads to Describe methods for review. Stage 2: Describe methods for review leads to Identify evidence: Animal human mechanistic. Stage 3: Identify evidence: Animal human mechanistic leads to Evaluate studies. Stage 4: Evaluate studies leads to Integrate evidence. Stage 5: Integrate evidence leads to Hazard Identification.
Figure 1.
Schematic for systematic review. Adapted from National Research Council (2014).
Figure 2 is a list of eight viewpoints under Bradford Hill, including Consistency, Specificity, Temporality, Biological gradient, Plausibility, Coherence, Experiment, and Analogy. A description is added to the right, stating “What I do not believe – and this has been suggested – is that we can usefully lay down some hard-and-fast rules of evidence that must be obeyed before we can accept cause and effect. None of my nine viewpoints can bring indisputable evidence for or against the cause-and-effect hypothesis and none can be required as a sine qua non. What they can do, with greater or less strength, is to help us make up our minds on the fundamental question – is there any other way of explaining the set of facts before us, is there any other answer equally, or more, likely than cause and effect?”
Figure 2.
Bradford Hill’s viewpoints (Hill 1965).
Figure 3 is a tabular representation of International Agency for Research on Cancer Monographs criteria for evidence synthesis, having four columns, namely, Evidence of cancer in humans, Evidence of cancer in experimental animals, Mechanistic evidence, and Evaluation. Row 1: Sufficient and Carcinogenic (Group 1). Row 2: Sufficient; Strong (exposed humans); and Carcinogenic (Group 1). Row 3: Limited; Sufficient; and Probably carcinogenic (Group 2 A). Row 4: Limited; Strong; and Probably carcinogenic (Group 2 A). Row 5: Sufficient; Strong (human cells or tissues); and Probably carcinogenic (Group 2 A). Row 6: Strong (mechanistic class) and Probably carcinogenic (Group 2 A). Row 7: Limited and Probably carcinogenic (Group 2 B). Row 8: Sufficient and Probably carcinogenic (Group 2 B). Row 9: Strong (experimental systems) and Probably carcinogenic (Group 2 B). Row 10: Sufficient; Strong (does not operate in humans); and Not classifiable (Group 3). Row 11: All other situations not listed here and Not classifiable (Group 3).
Figure 3.
IARC Monographs criteria for evidence synthesis. Adapted from IARC (2019b).
See this image and copyright information in PMC

References

    1. Angell M. 2008. Industry-sponsored clinical research: a broken system. JAMA 300(9):1069–1071, PMID: 18768418, 10.1001/jama.300.9.1069. - DOI - PubMed
    1. Armstrong BG. 1998. Effect of measurement error on epidemiological studies of environmental and occupational exposures. Occup Environ Med 55(10):651–656, PMID: 9930084, 10.1136/oem.55.10.651. - DOI - PMC - PubMed
    1. Arroyave WD, Mehta SS, Guha N, Schwingl P, Taylor KW, Glenn B, et al. . 2020. Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health. J Expo Sci Environ Epidemiol. In press, PMID: 32415298, 10.1038/s41370-020-0228-0. - DOI - PMC - PubMed
    1. Attfield MD, Schleiff PL, Lubin JH, Blair A, Stewart PA, Vermeulen R, et al. . 2012. The Diesel Exhaust in Miners study: a cohort mortality study with emphasis on lung cancer. J Natl Cancer Inst 104(11):869–883, PMID: 22393207, 10.1093/jnci/djs035. - DOI - PMC - PubMed
    1. Avanasi R, Shin HM, Vieira VM, Savitz DA, Bartell SM. 2016. Impact of exposure uncertainty on the association between perfluorooctanoate and preeclampsia in the C8 Health Project population. Environ Health Perspect 124(1):126–132, PMID: 26090912, 10.1289/ehp.1409044. - DOI - PMC - PubMed