Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011;6(7):e22998.
doi: 10.1371/journal.pone.0022998. Epub 2011 Jul 29.

Expert status and performance

Mark A Burgman  1 , Marissa McBride, Raquel Ashton, Andrew Speirs-Bridge, Louisa Flander, Bonnie Wintle, Fiona Fidler, Libby Rumpff, Charles Twardy
Affiliations

Expert status and performance

Mark A Burgman et al. PLoS One. 2011.

Abstract

Expert judgements are essential when time and resources are stretched or we face novel dilemmas requiring fast solutions. Good advice can save lives and large sums of money. Typically, experts are defined by their qualifications, track record and experience. The social expectation hypothesis argues that more highly regarded and more experienced experts will give better advice. We asked experts to predict how they will perform, and how their peers will perform, on sets of questions. The results indicate that the way experts regard each other is consistent, but unfortunately, ranks are a poor guide to actual performance. Expert advice will be more accurate if technical decisions routinely use broadly-defined expert groups, structured question protocols and feedback.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. Self assessment versus peer assessment of expertise.
Data from all workshops (overall correlation, r = 0.85). Peer assessment is the average of the scores on the 11-point scale provided by each person's peers on the day of the workshop. The strong relationship was consistent across the five groups, where the correlations ranged from 0.67 to 0.94 (Table 3). The dashed line is parity (where self assessment and peer assessment are equal).
Figure 2
Figure 2. Peer assessment of expert knowledge versus actual performance for the participants in Workshop 3.
Prediction accuracy is calculated as ALRE (see text). Small values for prediction accuracy are better. Closed circles and the solid line are estimates from round 1 (r = 0.19). Crosses and the dashed line are estimates from round 2 (r = −0.47). Estimates closer to the x-axis indicate the answers are closer to the truth.
Figure 3
Figure 3. The group average improvement in accuracy (ALRE) following discussion.
Change in estimates records distance from the truth, so that more strongly negative values improved more. The dots are the improvements in averages of best guesses. The error bars are 95% confidence intervals.
Figure 4
Figure 4. Accuracy of group means compared to highly regarded experts.
Mean and 95% confidence intervals of standardized distance from the truth (ALRE) for the most highly regarded individual in each workshop prior to discussion (‘Highest status’), and the workshop group average following discussion (‘Group Average’).
See this image and copyright information in PMC

References

    1. Hart A. NY: McGraw-Hill; 1986. Knowledge acquisition for expert systems.
    1. Collins HM, Evans R. Chicago: University of Chicago Press; 2007. Rethinking expertise.
    1. Barley SR, Kunda G. Contracting: a new form of professional practice. Academy of Management Perspective. 2006;20:45–66.
    1. Walton D. Pennsylvania: Pennsylvania State University Press; 1997. Appeal to expert opinion: arguments from authority.
    1. Morgan MG, Henrion M. Cambridge: Cambridge University Press, Cambridge; 1990. Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis.

Publication types