Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions

Andrew S Pullin^{1

2}, Samantha H Cheng³, Josephine D'Urban Jackson⁴, Jacqualyn Eales⁵, Ida Envall⁶, Salamatu J Fada^{7

8}, Geoff K Frampton⁹, Meagan Harper¹⁰, Andrew N Kadykalo¹⁰, Christian Kohl¹¹, Ko Konno¹², Barbara Livoreil¹³, Dakis-Yaoba Ouédraogo¹⁴, Bethan C O'Leary^{15

16}, George Pullin¹⁷, Nicola Randall¹⁸, Rebecca Rees¹⁹, Adrienne Smith²⁰, Romain Sordello²¹, Eleanor J Sterling²², Will M Twardek²³, Paul Woodcock²⁴

Affiliations

¹ Collaboration for Environmental Evidence, Conwy, UK. a.s.pullin@bangor.ac.uk.
² School of Natural Sciences, Bangor University, Bangor, UK. a.s.pullin@bangor.ac.uk.
³ Center for Biodiversity and Conservation, American Museum of Natural History, New York, USA.
⁴ Natural Resources Wales, Cambria House, 29 Newport Rd, Cardiff, CF24 0TP, UK.
⁵ European Centre for Environment and Human Health, College of Medicine and Health, University of Exeter, Knowledge Spa, Truro, TR1 3HD, UK.
⁶ The Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning, Formas, Stockholm, Sweden.
⁷ Bangor University, Bangor, UK.
⁸ University of Jos, Jos, Nigeria.
⁹ Southampton Health Technology Assessments Centre, Faculty of Medicine, University of Southampton, Southampton, UK.
¹⁰ Canadian Centre for Evidence-Based Conservation, Department of Biology and Institute of Environmental and Interdisciplinary Science, Carleton University, Ottawa, Canada.
¹¹ Julius Kühn Institute (JKI), Federal Research Centre for Cultivated Plants, Institute for Biosafety in Plant Biotechnology (SB), Quedlinburg, Germany.
¹² School of Natural Sciences, Bangor University, Bangor, UK.
¹³ , 30 rue Lamartine, 83340, Le Luc, France.
¹⁴ Direction de L'Expertise, Muséum National d'Histoire Naturelle (MNHN), 75005, Paris, France.
¹⁵ Centre for Ecology & Conservation, College of Life and Environmental Sciences, University of Exeter, Penryn Campus, Penryn, UK.
¹⁶ Department of Environment and Geography, University of York, York, UK.
¹⁷ University of York, York, UK.
¹⁸ Centre for Evidence Based Agriculture, Harper Adams University, Newport, UK.
¹⁹ EPPI-Centre, UCL Social Research Institute, University College London, London, UK.
²⁰ Canadian Centre for Evidence-Based Conservation, Carleton University, Ottawa, Canada.
²¹ UMS PatriNat OFB-CNRS-MNHN, Paris, France.
²² Center for Biodiversity and Conservation, American Museum of Natural History, New York, NY, 10024, USA.
²³ Fish Ecology and Conservation Physiology Lab, Carleton University, 1125 Colonel By Dr, Ottawa, ON, K1S 5B6, Canada.
²⁴ Joint Nature Conservation Committee, Monkstone House, Peterborough, PE1 1JY, UK.

PMID: 39294776
PMCID: PMC11378768
DOI: 10.1186/s13750-022-00269-9

Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions

Andrew S Pullin et al. Environ Evid. 2022.

. 2022 Apr 19;11(1):16.

doi: 10.1186/s13750-022-00269-9.

Authors

Affiliations

¹ Collaboration for Environmental Evidence, Conwy, UK. a.s.pullin@bangor.ac.uk.
² School of Natural Sciences, Bangor University, Bangor, UK. a.s.pullin@bangor.ac.uk.
³ Center for Biodiversity and Conservation, American Museum of Natural History, New York, USA.
⁴ Natural Resources Wales, Cambria House, 29 Newport Rd, Cardiff, CF24 0TP, UK.
⁵ European Centre for Environment and Human Health, College of Medicine and Health, University of Exeter, Knowledge Spa, Truro, TR1 3HD, UK.
⁶ The Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning, Formas, Stockholm, Sweden.
⁷ Bangor University, Bangor, UK.
⁸ University of Jos, Jos, Nigeria.
⁹ Southampton Health Technology Assessments Centre, Faculty of Medicine, University of Southampton, Southampton, UK.
¹⁰ Canadian Centre for Evidence-Based Conservation, Department of Biology and Institute of Environmental and Interdisciplinary Science, Carleton University, Ottawa, Canada.
¹¹ Julius Kühn Institute (JKI), Federal Research Centre for Cultivated Plants, Institute for Biosafety in Plant Biotechnology (SB), Quedlinburg, Germany.
¹² School of Natural Sciences, Bangor University, Bangor, UK.
¹³ , 30 rue Lamartine, 83340, Le Luc, France.
¹⁴ Direction de L'Expertise, Muséum National d'Histoire Naturelle (MNHN), 75005, Paris, France.
¹⁵ Centre for Ecology & Conservation, College of Life and Environmental Sciences, University of Exeter, Penryn Campus, Penryn, UK.
¹⁶ Department of Environment and Geography, University of York, York, UK.
¹⁷ University of York, York, UK.
¹⁸ Centre for Evidence Based Agriculture, Harper Adams University, Newport, UK.
¹⁹ EPPI-Centre, UCL Social Research Institute, University College London, London, UK.
²⁰ Canadian Centre for Evidence-Based Conservation, Carleton University, Ottawa, Canada.
²¹ UMS PatriNat OFB-CNRS-MNHN, Paris, France.
²² Center for Biodiversity and Conservation, American Museum of Natural History, New York, NY, 10024, USA.
²³ Fish Ecology and Conservation Physiology Lab, Carleton University, 1125 Colonel By Dr, Ottawa, ON, K1S 5B6, Canada.
²⁴ Joint Nature Conservation Committee, Monkstone House, Peterborough, PE1 1JY, UK.

PMID: 39294776
PMCID: PMC11378768
DOI: 10.1186/s13750-022-00269-9

Abstract

Accurate, unbiased and concise synthesis of available evidence following clear methodology and transparent reporting is necessary to support effective environmental policy and management decisions. Without this, less reliable and/or less objective reviews of evidence could inform decision making, leading to ineffective, resource wasteful interventions with potential for unintended consequences. We evaluated the reliability of over 1000 evidence syntheses (reviews and overviews) published between 2018 and 2020 that provide evidence on the impacts of human activities or effectiveness of interventions relevant to environmental management. The syntheses are drawn from the Collaboration for Environmental Evidence Database of Evidence Reviews (CEEDER), an online, freely available evidence service for evidence users that assesses the reliability of evidence syntheses using a series of published criteria. We found that the majority of syntheses have problems with transparency, replicability and potential for bias. Overall, our results suggest that most recently published evidence syntheses are of low reliability to inform decision making. Reviews that followed guidance and reporting standards for evidence synthesis had improved assessment ratings, but there remains substantial variation in the standard of reviews amongst even these. Furthermore, the term 'systematic review', which implies conformity with a methodological standard, was frequently misused. A major objective of the CEEDER project is to improve the reliability of the global body of environmental evidence reviews. To this end we outline freely available online resources to help improve review conduct and reporting. We call on authors, editors and peer reviewers to use these resources to ensure more reliable syntheses in the future.

Keywords: CEEDER; CEESAT; Environmental evidence; Evidence synthesis; Evidence-informed decision making; Review reliability.

PubMed Disclaimer

Conflict of interest statement

All authors of this article contribute to CEEDER voluntarily. We declare no competing interests.

Figures

**Fig. 1**
Number of evidence syntheses indexed in the CEEDER database by country of origin (corresponding authors’ affiliation). Some syntheses are counted multiple times as authors may be affiliated with multiple institutions located in different countries. Frequencies are provided for those that exceed 30 (China, USA, Australia, UK, Canada, Germany, Brazil and India)

**Fig. 2**
The distribution of CEESAT ratings for each criterion for evidence reviews (n = 924, top) and evidence overviews (n = 134, bottom) published between 2018 and 2020. Note, no red category is included for Criterion 1 as this is an eligibility criterion for inclusion in the CEEDER database (red articles for criterion 1 are excluded from CEEDER). CEESAT criteria 5 and 7 are not applied to overviews

**Fig. 3**
Distribution of assessment ratings in a evidence reviews citing CEE Guidelines (including CEE Systematic Reviews) n = 23/924; b evidence reviews citing PRISMA reporting checklist n = 117/924; and c evidence reviews where authors claim to have conducted a Systematic Review but were not registered by CEE n = 90/924. X axis shows absolute numbers of evidence reviews

See this image and copyright information in PMC

References

1. Christie AP, Abecasis D, Adjeroud M. Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nat Commun. 2020;11:6377. 10.1038/s41467-020-20142-y - DOI - PMC - PubMed
1. Collaboration for Environmental Evidence. Guidelines and standards for evidence synthesis in environmental management. version 5.0 (AS Pullin, GK Frampton, B Livoreil & G Petrokofsky, Eds). 2018. www.environmentalevidence.org/information-for-authors.
1. Haddaway NR, Pullin AS. The policy role of systematic reviews: past, present and future. Springer Sci Rev. 2014;2:179–83. 10.1007/s40362-014-0023-1 - DOI
1. Haddaway NR, Woodcock P, Macura B, Collins A. Making literature reviews more reliable through application of lessons from systematic reviews. Conserv Biol. 2015;29:1596–605. 10.1111/cobi.12541 - DOI - PubMed
1. Haddaway NR, Macura B, Whalley P, Pullin AS. ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ Evid. 2018;7:7. 10.1186/s13750-018-0121-7 - DOI

LinkOut - more resources

Full Text Sources
- PubMed Central
Miscellaneous
- NCI CPTAC Assay Portal

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions

Affiliations

Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

LinkOut - more resources

Full Text Sources

Miscellaneous