Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Apr 22;17(4):e0266125.
doi: 10.1371/journal.pone.0266125. eCollection 2022.

Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: A scoping review

Grace Yeeun Lee  1 , Ian Bernard Hickie  1 , Jo-An Occhipinti  1   2 , Yun Ju Christine Song  1 , Adam Skinner  1 , Salvador Camacho  3   4 , Kenny Lawson  1 , Adriane Martin Hilber  3   4 , Louise Freebairn  1   2   5
Affiliations

Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: A scoping review

Grace Yeeun Lee et al. PLoS One. .

Abstract

Introduction: Systems modelling and simulation can improve understanding of complex systems to support decision making, better managing system challenges. Advances in technology have facilitated accessibility of modelling by diverse stakeholders, allowing them to engage with and contribute to the development of systems models (participatory modelling). However, despite its increasing applications across a range of disciplines, there is a growing need to improve evaluation efforts to effectively report on the quality, importance, and value of participatory modelling. This paper aims to identify and assess evaluation frameworks, criteria, and/or processes, as well as to synthesize the findings into a comprehensive multi-scale framework for participatory modelling programs.

Materials and methods: A scoping review approach was utilized, which involved a systematic literature search via Scopus in consultation with experts to identify and appraise records that described an evaluation framework, criteria, and/or process in the context of participatory modelling. This scoping review is registered with the Open Science Framework.

Results: The review identified 11 studies, which varied in evaluation purposes, terminologies, levels of examination, and time points. The review of studies highlighted areas of overlap and opportunities for further development, which prompted the development of a comprehensive multi-scale evaluation framework to assess participatory modelling programs across disciplines and systems modelling methods. The framework consists of four categories (Feasibility, Value, Change/Action, Sustainability) with 30 evaluation criteria, broken down across project-, individual-, group- and system-level impacts.

Discussion & conclusion: The presented novel framework brings together a significant knowledge base into a flexible, cross-sectoral evaluation effort that considers the whole participatory modelling process. Developed through the rigorous synthesis of multidisciplinary expertise from existing studies, the application of the framework can provide the opportunity to understand practical future implications such as which aspects are particularly important for policy decisions, community learning, and the ongoing improvement of participatory modelling methods.

PubMed Disclaimer

Conflict of interest statement

Dr Louise Freebairn is currently employed part-time by the Brain and Mind Centre, University of Sydney; ACT Health as Director of Knowledge Translation and Health Outcomes, Epidemiology Section, ACT Health & CSART as Director of Policy Applications & Translational Science. A/Professor Jo-An Occhipinti is both Head of Systems Modelling, Simulation & Data Science at the University of Sydney’s Brain and Mind Centre and Managing Director of Computer Simulation & Advanced Research Technologies (CSART). Professor Ian Hickie is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC) University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to headspace. He is the Chief Scientific Advisor to, and a 5% equity shareholder in, InnoWell Pty Ltd. InnoWell was formed by the University of Sydney (45% equity) and PwC (Australia; 45% equity) to deliver the $30 M Australian Government-funded Project Synergy (2017-20; a three-year program for the transformation of mental health services) and to lead transformation of mental health services internationally through the use of innovative technologies.

Figures

Fig 1
Fig 1. PRISMA flow diagram.
Fig 2
Fig 2. Word cloud generated by uploading the full text of all studies included for synthesis.
Fig 3
Fig 3. Comprehensive multi-scale evaluation framework for PM programs.
Fig 4
Fig 4. Evaluation framework underpinned by participatory action research, with consideration of impacts across the project-, individual-, group-, and system-level.
Fig 5
Fig 5. Potential methods of data collection to adopt the comprehensive multi-scale framework into evaluation practice.
See this image and copyright information in PMC

References

    1. AF Bala BK., Noh KM. System Dynamics: Modelling and Simulation. Singapore: Springer Nature; 2017.
    1. Kelly RA, Jakeman AJ, Barreteau O, Borsuk ME, ElSawah S, Hamilton SH, et al.. Selecting among five common modelling approaches for integrated environmental assessment and management. Environmental Modelling & Software. 2013;47:159–81. 10.1016/j.envsoft.2013.05.005. - DOI
    1. A B. The Big Book of Simulation Modeling: Multimethod Modeling with AnyLogic 6. Chicago: AnyLogic North America; 2013.
    1. Currie DJ, Smith C, Jagals P. The application of system dynamics modelling to environmental health decision-making and policy—a scoping review. BMC Public Health. 2018;18(1):402. doi: 10.1186/s12889-018-5318-8 - DOI - PMC - PubMed
    1. Freebairn L, Atkinson J-A, Kelly PM, McDonnell G, Rychetnik L. Decision makers’ experience of participatory dynamic simulation modelling: methods for public health policy. BMC Medical Informatics and Decision Making. 2018;18(1):131. doi: 10.1186/s12911-018-0707-6 - DOI - PMC - PubMed

Publication types