The Wisdom of Crowds of Doctors: Their Average Predictions Outperform Their Individual Ones

Abstract

Background: Evidence suggests that the average prediction across groups is more accurate than for individuals. Our goals were therefore to investigate accuracy of the average predictions for groups of clinicians and to compare this accuracy with a published statistical prediction model.

Methods: Twenty-four expert clinicians attending an advisory board meeting were asked to make predictions for 25 patients from a research registry regarding the probability of having a positive bone scan 1 year from today if left untreated. Comparisons were made between the accuracy of average responses and that of an appropriate previously published statistical prediction model.

Results: This study suggests that the mean of the clinicians' predictions can quickly approach the accuracy of the best clinician using as few as 5 clinicians. When all 24 clinicians' predictions were averaged, the concordance index reached 0.750, still far below that of the published statistical model with 0.812.

Conclusions: Averaging clinician predictions may have merit over individual clinician predictions but still not reasonably replace a carefully built statistical model. However, averaging clinician predictions could prove helpful in situations where statistical models do not yet exist or where existing models are inadequate.

Keywords: C-statistic; accuracy; nomogram; prediction; wisdom of crowds.