An evaluation of clinicians' subjective prior probability estimates

Abstract

The degree of consensus and the accuracy of subjective prior probability estimates made by 104 clinicians were examined. The clinicians' estimates were compared with objective prior probabilities obtained from published sources and actual patient outcomes. Each clinician made seven estimates based upon written case summaries abstracted from patient records. Consensus was measured by calculating estimate ranges and standard deviations. The clinicians' estimates varied widely: the smallest range was 80 (2%-82%); four of the seven probability ranges were greater than 90. The average standard deviation was 19.5. Using these prior probabilities and Bayes' theorem, widely varying posttest probabilities would result after many common diagnostic tests. Accuracy was measured using the Brier score, which ranges from 0 to 1; a score of 0 indicates perfect accuracy. The clinicians' Brier scores ranged from 0.05 to 0.57. The objectively determined probabilities achieved a Brier score of 0.11, better than that of 96% of the clinicians. Clinical experience did not consistently affect estimate accuracy or consensus. The clinicians' subjective estimates were inaccurate measures of the prior probability of disease. There was little consensus regarding disease likelihood among the clinicians. Objective prior probabilities were more accurate and less variable.