Validation of consensus panel diagnosis in dementia

Abstract

Background: The clinical diagnosis of dementing diseases largely depends on the subjective interpretation of patient symptoms. Consensus panels are frequently used in research to determine diagnoses when definitive pathologic findings are unavailable. Nevertheless, research on group decision making indicates that many factors can adversely affect panel performance.

Objective: To determine conditions that improve consensus panel diagnosis.

Design: Comparison of neuropathologic diagnoses with individual and consensus panel diagnoses based on clinical scenarios only, fludeoxyglucose F 18 positron emission tomography images only, and scenarios plus images.

Setting: Expert and trainee individual and consensus panel deliberations using a modified Delphi method in a pilot research study of the diagnostic utility of fludeoxyglucose F 18 positron emission tomography.

Patients: Forty-five patients with pathologically confirmed Alzheimer disease or frontotemporal dementia.

Main outcome measures: Statistical measures of diagnostic accuracy, agreement, and confidence for individual raters and panelists before and after consensus deliberations.

Results: The consensus protocol using trainees and experts surpassed the accuracy of individual expert diagnoses when clinical information elicited diverse judgments. In these situations, consensus was 3.5 times more likely to produce positive rather than negative changes in the accuracy and diagnostic certainty of individual panelists. A rule that forced group consensus was at least as accurate as majority and unanimity rules.

Conclusions: Using a modified Delphi protocol to arrive at a consensus diagnosis is a reasonable substitute for pathologic information. This protocol improves diagnostic accuracy and certainty when panelist judgments differ and is easily adapted to other research and clinical settings while avoiding the potential pitfalls of group decision making.

Figure 1

Comparison of diagnostic accuracy of expert raters, and panelists with expert panels reviewing clinical scenarios Consensus panel diagnoses (in black) based on scenarios were more accurate than diagnoses arrived at by 10 of 11 individual panelists and 5 of 6 raters. This panel superiority was statistically significant (in red) for all members of the trainee panel, 4 of members of the expert panel, and 3 of the raters (p<.05, Hochberg corrected). Note that one member of the trainee panel did not review 17 cases and was thus omitted from this analysis.

Figure 2

Comparison of diagnostic accuracy of expert raters and panelists with panels reviewing FDG-PET images Consensus panel diagnoses (in black) based on images alone or with clinical scenarios were more accurate than diagnoses of 0 of 6 panelists and 2 of 6 raters when reviewing images and scenarios (A) and 2 of 6 panelists and 2 of 6 raters when reviewing images alone (B). Two panelists and two raters (in red) had a statistically significant lower accuracy than each panel (p<.05, Hochberg corrected). In contrast, across the two panels, 6 of 12 panelists and 5 of 12 raters (in blue) had a statistically significant greater accuracy than the panel diagnoses (p<.,05, Hochberg corrected).

Figure 3

Panel diagnostic accuracy by patient Each horizontal line represents a single patient. Panel diagnoses in agreement with neuropathological diagnoses are shown in gray. Panel diagnostic errors are in red. Panels often were in error in the same patients. The pairwise Kappa agreement between diagnoses of trainee and expert panel for scenarios was 0.79 (±0.15 SE); trainee (scenario) and expert (images) panels was 0.68 (±.15 SE); trainee (scenario) and expert (scenario+images) was .79 (±.15 SE); expert (scenario) and expert (images) panels was .69 (±.15 SE); expert (images) and expert (scenario+images) panels was .79 (±.15 SE).