Randomized Controlled Trial

. 2023 Jul 19;24(1):461.

doi: 10.1186/s13063-023-07388-9.

Multi-Round versus Real-Time Delphi survey approach for achieving consensus in the COHESION core outcome set: a randomised trial

Fiona A Quirke^{1

2

3}, Malcolm R Battin⁴, Caitlin Bernard⁵, Linda Biesty^{6

7

8}, Frank H Bloomfield⁹, Mandy Daly¹⁰, Elaine Finucane⁷, David M Haas⁵, Patricia Healy^{6

7}, Tim Hurley^{11

12}, Sarah Koskei¹³, Shireen Meher¹⁴, Eleanor J Molloy^{11

15

16}, Maira Niaz¹⁷, Elaine Ní Bhraonáin¹⁸, Christabell Omukagah Okaronon¹⁹, Farhana Tabassum²⁰, Karen Walker^{21

22

23

24}, James R H Webbe²⁵, Matthew J Parkes²⁶, Jamie J Kirkham²⁷, Declan Devane^{11

28

6

29}

Affiliations

¹ Health Research Board Neonatal Encephalopathy PhD Training Network (NEPTuNE), Dublin, Ireland. f.quirke1@nuigalway.ie.
² Health Research Board - Trials Methodology Research Network (HRB-TMRN), Galway, Ireland. f.quirke1@nuigalway.ie.
³ School of Nursing and Midwifery, University of Galway, Galway, Ireland. f.quirke1@nuigalway.ie.
⁴ Department of Newborn Services, Auckland District Health Board, Auckland, New Zealand.
⁵ Department of Obstetrics and Gynecology, Indiana University, Indianapolis, IN, USA.
⁶ School of Nursing and Midwifery, University of Galway, Galway, Ireland.
⁷ Evidence Synthesis Ireland, University of Galway, Galway, Ireland.
⁸ Qualitative Research in Trials Centre (QUESTS), University of Galway, Galway, Ireland.
⁹ Liggins Institute, University of Auckland, Private Bag 92019, Auckland, 1142, New Zealand.
¹⁰ Advocacy and Policymaking, Irish Neonatal Health Alliance, Wicklow, Ireland.
¹¹ Health Research Board Neonatal Encephalopathy PhD Training Network (NEPTuNE), Dublin, Ireland.
¹² Department of Paediatric and Child Health, Trinity College Dublin, Tallaght University Hospital (TUH), Dublin, Ireland.
¹³ Moi University, Eldoret, Kenya.
¹⁴ Birmingham Women's and Children's NHS Foundation Trust, Birmingham, UK.
¹⁵ Paediatrics and Child Health, Trinity College Dublin, Dublin, Ireland.
¹⁶ Department of Neonatology, Children's Hospital Ireland at Crumlin and Tallaght, Coombe Women and Infants University Hospital, Dublin, Ireland.
¹⁷ Department of Paediatrics & Child Health, Aga Khan University, Karachi, Pakistan.
¹⁸ Family Support Liaison, Irish Neonatal Health Alliance, Wicklow, Ireland.
¹⁹ AMPATH, Eldoret, Kenya.
²⁰ Centre of Excellence in Women and Child Health, Aga Khan University, Karachi, Pakistan.
²¹ Department of Newborn Care, Royal Prince Alfred Hospital, Sydney, Australia.
²² University of Sydney, Sydney, Australia.
²³ The George Institute for Global Health, Sydney, Australia.
²⁴ Council of International Neonatal Nurses, Sydney, Australia.
²⁵ Academic Neonatal Medicine, Imperial College London, London, UK.
²⁶ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics Rheumatology and Musculoskeletal Science, University of Oxford, Oxfordshire, UK.
²⁷ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
²⁸ Health Research Board - Trials Methodology Research Network (HRB-TMRN), Galway, Ireland.
²⁹ Cochrane Ireland, University of Galway, Galway, Ireland.

PMID: 37468987
PMCID: PMC10354951
DOI: 10.1186/s13063-023-07388-9

Randomized Controlled Trial

Multi-Round versus Real-Time Delphi survey approach for achieving consensus in the COHESION core outcome set: a randomised trial

Fiona A Quirke et al. Trials. 2023.

. 2023 Jul 19;24(1):461.

doi: 10.1186/s13063-023-07388-9.

Authors

Affiliations

¹ Health Research Board Neonatal Encephalopathy PhD Training Network (NEPTuNE), Dublin, Ireland. f.quirke1@nuigalway.ie.
² Health Research Board - Trials Methodology Research Network (HRB-TMRN), Galway, Ireland. f.quirke1@nuigalway.ie.
³ School of Nursing and Midwifery, University of Galway, Galway, Ireland. f.quirke1@nuigalway.ie.
⁴ Department of Newborn Services, Auckland District Health Board, Auckland, New Zealand.
⁵ Department of Obstetrics and Gynecology, Indiana University, Indianapolis, IN, USA.
⁶ School of Nursing and Midwifery, University of Galway, Galway, Ireland.
⁷ Evidence Synthesis Ireland, University of Galway, Galway, Ireland.
⁸ Qualitative Research in Trials Centre (QUESTS), University of Galway, Galway, Ireland.
⁹ Liggins Institute, University of Auckland, Private Bag 92019, Auckland, 1142, New Zealand.
¹⁰ Advocacy and Policymaking, Irish Neonatal Health Alliance, Wicklow, Ireland.
¹¹ Health Research Board Neonatal Encephalopathy PhD Training Network (NEPTuNE), Dublin, Ireland.
¹² Department of Paediatric and Child Health, Trinity College Dublin, Tallaght University Hospital (TUH), Dublin, Ireland.
¹³ Moi University, Eldoret, Kenya.
¹⁴ Birmingham Women's and Children's NHS Foundation Trust, Birmingham, UK.
¹⁵ Paediatrics and Child Health, Trinity College Dublin, Dublin, Ireland.
¹⁶ Department of Neonatology, Children's Hospital Ireland at Crumlin and Tallaght, Coombe Women and Infants University Hospital, Dublin, Ireland.
¹⁷ Department of Paediatrics & Child Health, Aga Khan University, Karachi, Pakistan.
¹⁸ Family Support Liaison, Irish Neonatal Health Alliance, Wicklow, Ireland.
¹⁹ AMPATH, Eldoret, Kenya.
²⁰ Centre of Excellence in Women and Child Health, Aga Khan University, Karachi, Pakistan.
²¹ Department of Newborn Care, Royal Prince Alfred Hospital, Sydney, Australia.
²² University of Sydney, Sydney, Australia.
²³ The George Institute for Global Health, Sydney, Australia.
²⁴ Council of International Neonatal Nurses, Sydney, Australia.
²⁵ Academic Neonatal Medicine, Imperial College London, London, UK.
²⁶ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics Rheumatology and Musculoskeletal Science, University of Oxford, Oxfordshire, UK.
²⁷ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
²⁸ Health Research Board - Trials Methodology Research Network (HRB-TMRN), Galway, Ireland.
²⁹ Cochrane Ireland, University of Galway, Galway, Ireland.

PMID: 37468987
PMCID: PMC10354951
DOI: 10.1186/s13063-023-07388-9

Abstract

Background: Delphi surveys are commonly used to prioritise critical outcomes in core outcome set (COS) development. This trial aims to compare a three-round (Multi-Round) Delphi (MRD) with a Real-Time Delphi (RTD) in the prioritisation of outcomes for inclusion in a COS for neonatal encephalopathy treatments and explore whether 'feedback', 'iteration', and 'initial condition' effects may occur in the two survey methods.

Methods: We recruited 269 participants (parents/caregivers, healthcare providers and researchers/academics) of which 222 were randomised to either the MRD or the RTD. We investigated the outcomes prioritised in each survey and the 'feedback', 'iteration', and 'initial condition' effects to identify differences between the two survey methods.

Results: In the RTD, n = 92 participants (83%) fully completed the survey. In the MRD, n = 60 participants (54%) completed all three rounds. Of the 92 outcomes presented, 26 (28%) were prioritised differently between the RTD and MRD. Significantly fewer participants amended their scores when shown stakeholder responses in the RTD compared to the MRD ('feedback effect'). The 'iteration effect' analysis found most experts appeared satisfied with their initial ratings in the RTD and did not amend their scores following stakeholder response feedback. Where they did amend their scores, ratings were amended substantially, suggesting greater convergence. Variance in scores reduced with subsequent rounds of the MRD ('iteration effect'). Whilst most participants did not change their initial scores in the RTD, of those that did, later recruits tended to align their final score more closely to the group mean final score than earlier recruits (an 'initial condition' effect).

Conclusion: The feedback effect differed between the two Delphi methods but the magnitude of this difference was small and likely due to the large number of observations rather than because of a meaningfully large difference. It did not appear to be advantageous to require participants to engage in three rounds of a survey due to the low change in scores. Larger drop-out through successive rounds in the MRD, together with a lesser convergence of scores and longer time to completion, indicate considerable benefits of the RTD approach.

Trial registration: NCT04471103. Registered on 14 July 2020.

Keywords: Consensus; Core outcome set; Delphi survey; Multi-round Delphi; Randomised trial; Real-Time Delphi.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

**Fig. 2**
Example of stakeholder rating responses in bar chart format

**Fig. 3**
CONSORT Flow Diagram of the progress through the phases of a parallel randomised trial of two groups

**Fig. 4**
‘Feedback effect’ model indicating slopes of the difference between initial and final rates of participants in the two survey methods

**Fig. 5**
Histograms showing the distribution of the difference between initial and final scores in all participants in both survey types

**Fig. 6**
Histograms showing the distribution of the difference between initial and final scores in those participants that revisited their initial rating at least once

**Fig. 7**
Comparison of the difference (deviation) between initial score of participants and the average final score, and the difference between initial and final scores, for those participants that who altered their score between initial and final rating only. Plotted lines are slopes taken from mixed effects regression model

See this image and copyright information in PMC

Cited by

From athletic excellence to academic influence: a study of retired Chinese athletes transitioning into the higher education sector.
Zhou Y, Zhang Z. Zhou Y, et al. Front Psychol. 2024 Jun 18;15:1401575. doi: 10.3389/fpsyg.2024.1401575. eCollection 2024. Front Psychol. 2024. PMID: 38957888 Free PMC article.
Surgical Handover Core Outcome Measures (SH-CORE): a protocol for the development of a core outcome set for trials in surgical handover.
Ryan JM, Devane D, Simiceva A, Eppich W, Kavanagh DO, Cullen C, Hogan AM, McNamara DA. Ryan JM, et al. Trials. 2024 Jun 10;25(1):373. doi: 10.1186/s13063-024-08201-x. Trials. 2024. PMID: 38858749 Free PMC article.
Protocol for the development of a core outcome set for respectful maternal and newborn care in a low-middle income setting.
Marenga F, Dube K, Goshomi U, Lavender T, Bedwell C, Kirkham JJ. Marenga F, et al. PLoS One. 2025 Apr 9;20(4):e0319419. doi: 10.1371/journal.pone.0319419. eCollection 2025. PLoS One. 2025. PMID: 40203012 Free PMC article.
How Delphi studies in the health sciences find consensus: a scoping review.
Schifano J, Niederberger M. Schifano J, et al. Syst Rev. 2025 Jan 14;14(1):14. doi: 10.1186/s13643-024-02738-3. Syst Rev. 2025. PMID: 39810238 Free PMC article.
Consensus definition and diagnostic criteria for neonatal encephalopathy-study protocol for a real-time modified delphi study.
Branagan A, Hurley T, Quirke F, Devane D, Taneri PE, Badawi N, Sinha B, Bearer C, Bloomfield FH, Bonifacio SL, Boylan G, Campbell SK, Chalak L, D'Alton M, deVries LS, El Dib M, Ferriero DM, Gale C, Gressens P, Gunn AJ, Kay S, Maeso B, Mulkey SB, Murray DM, Nelson KB, Nesterenko TH, Pilon B, Robertson NJ, Walker K, Wusthoff CJ, Molloy EJ; Steering Group for DEFiNE (Definition of Neonatal Encephalopathy). Branagan A, et al. Pediatr Res. 2025 Jan;97(1):430-436. doi: 10.1038/s41390-024-03303-3. Epub 2024 Jun 20. Pediatr Res. 2025. PMID: 38902453 Free PMC article.

See all "Cited by" articles

References

1. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(3):280. doi: 10.1186/s13063-017-1978-4. - DOI - PMC - PubMed
1. Biggane AM, Williamson PR, Ravaud P, Young B. Participating in core outcome set development via Delphi surveys: qualitative interviews provide pointers to inform guidance. BMJ Open. 2019;9(11):e032338. doi: 10.1136/bmjopen-2019-032338. - DOI - PMC - PubMed
1. Dalkey N, Helmer O. An experimental application of the Delphi method the use of experts. Manag Sci. 1963;9:458–67. doi: 10.1287/mnsc.9.3.458. - DOI
1. Rowe G, Wright G, Bolger F. Delphi: a reevaluation of research and theory. Technol Forecast Soc Chang. 1991;39(3):235–251. doi: 10.1016/0040-1625(91)90039-I. - DOI
1. Hall DA, Smith H, Heffernan E, Fackrell K. Recruiting and retaining participants in e-Delphi surveys for core outcome set development: evaluating the COMiT’ ID study. PLoS One. 2018;13(7):e0201378. doi: 10.1371/journal.pone.0201378. - DOI - PMC - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Associated data

Actions
- Search in PubMed
- Search in ClinicalTrials.gov

Grants and funding

CDA-FA-2018-008/HRBI_/Health Research Board/Ireland

LinkOut - more resources

Full Text Sources
Medical
- ClinicalTrials.gov

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Multi-Round versus Real-Time Delphi survey approach for achieving consensus in the COHESION core outcome set: a randomised trial

Affiliations

Multi-Round versus Real-Time Delphi survey approach for achieving consensus in the COHESION core outcome set: a randomised trial

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Associated data

Grants and funding

LinkOut - more resources

Full Text Sources

Medical

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Associated data

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Medical