Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Jul;42(5):684-703.
doi: 10.1177/0272989X211053794. Epub 2021 Oct 25.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

Christopher J Cadham  1 , Marie Knoll  2 , Luz María Sánchez-Romero  2 , K Michael Cummings  3 , Clifford E Douglas  1   4 , Alex Liber  2 , David Mendez  1 , Rafael Meza  5 , Ritesh Mistry  6 , Aylin Sertkaya  7 , Nargiz Travis  1   2 , David T Levy  2
Affiliations

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

Christopher J Cadham et al. Med Decis Making. 2022 Jul.

Abstract

Background: Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

Objectives: We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

Data sources: PubMed and Web of Science.

Study eligibility: Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

Data abstraction and synthesis: Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

Study appraisal: Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

Results: The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

Limitations: Elicitations on nonhealth topics and those reported in the gray literature were not included.

Conclusions: We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

Highlights: We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

Keywords: computational modeling; expert elicitation; expert judgment; expert opinion; systematic review.

PubMed Disclaimer

Conflict of interest statement

KMC has served on advisory committees for Pfizer, Inc. to assist them in ways to promote access to smoking cessation treatments. He has also served as a paid expert witness in litigation filed against the tobacco industry.

The other authors report no conflicts of interest.

Figures

Figure 1
Figure 1
PRISMA diagram of identified studies of Expert Elicitations conducted in health services research-based modeling studies.
See this image and copyright information in PMC

References

    1. Maglio PP, Sepulveda MJ, Mabry PL. Mainstreaming modeling and simulation to accelerate public health innovation. Am J Public Health. 2014;104(7):1181–1186. - PMC - PubMed
    1. O’Donnell E, Atkinson JA, Freebairn L, Rychetnik L. Participatory simulation modelling to inform public health policy and practice: Rethinking the evidence hierarchies. J Public Health Policy. 2017;38(2):203–215. - PubMed
    1. Petitti DB, Lin JS, Owens DK, Croswell JM, Feuer EJ. Collaborative Modeling: Experience of the U.S. Preventive Services Task Force. Am J Prev Med. 2018;54(1s1):S53–s62. - PubMed
    1. Habbema JD, Wilt TJ, Etzioni R, et al. Models in the development of clinical practice guidelines. Ann Intern Med. 2014;161(11):812–818. - PubMed
    1. Morgan MG. Use (and abuse) of expert elicitation in support of decision making for public policy. Proc Natl Acad Sci U S A. 2014;111(20):7176–7184. - PMC - PubMed

Publication types