A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement

Abstract

Objectives: The objective of the study was to explore the impact of different feedback strategies on (1) subsequent agreement and (2) variability in Delphi studies.

Study design and setting: A two-round Delphi survey, with a list of outcomes generated from the results of a systematic review and interviews, was undertaken while developing a core outcomes set for prostate cancer including two stakeholder groups (health professionals and patients). Seventy-nine outcomes were scored on a scale of one (not important) to nine (critically important). Participants were randomized in round 2 to receive round 1 feedback from peers only, multiple stakeholders separately, or multiple stakeholders combined.

Results: Agreement on outcomes retained for all feedback groups was high (peer: 92%, multiple separate: 90%, multiple combined: 84%). There were no statistically significant reduction in variability for peer vs. multiple separate (0.016 [-0.035, 0.067]; P = 0.529), or multiple separate vs. multiple combined feedback (0.063 [-0.003, 0.129]; P = 0.062). Peer feedback statistically significantly reduced variability compared with multiple combined feedback (0.079 [0.001, 0.157]; P = 0.046).

Conclusions: We found no evidence of a difference between different feedback strategies in terms of the number of outcomes retained or reduction in variability of opinion. However, this may be explained by the high level of existing agreement in round 1. Further methodological studies nested within Delphi surveys will help clarify the best strategy.

Keywords: Consensus methods; Core outcome set development; Delphi study; Feedback strategies; RCT; Stakeholders.