360-degree Delphi: addressing sociotechnical challenges of healthcare IT

Abstract

Background: IT systems in the healthcare field can have a marked sociotechnical impact: they modify communication habits, alter clinical processes and may have serious ethical implications. The introduction of such systems involves very different groups of stakeholders because of the inherent multi-professionalism in medicine and the role of patients and their relatives that are often underrepresented. Each group contributes distinct perspectives and particular needs, which create specific requirements for IT systems and may strongly influence their acceptance and success. In the past, needs analysis, challenges and requirements for medical IT systems have often been addressed using consensus techniques such as the Delphi technique. Facing the heterogeneous spectrum of stakeholders there is a need to develop these techniques further to control the (strong) influence of the composition of the expert panel on the outcome and to deal systematically with potentially incompatible needs of stakeholder groups. This approach uses the strong advantages a Delphi study has, identifies the disadvantages of traditional Delphi techniques and aims to introduce and evaluate a modified approach called 360-Degree Delphi. Key aspects of 360-Degree Delphi are tested by applying the approach to the needs and requirements analysis of a system for managing patients' advance directives and living wills.

Methods: 360-Degree Delphi (short 360°D), as a modified Delphi process, is specified as a structured workflow with the optional use of stakeholder groups. The approach redefines the composition of the expert panel by setting up groups of different stakeholders. Consensus is created within individual stakeholder groups, but is also communicated between groups, while the iterative structure of the Delphi process remains unchanged. We hypothesize that (1) 360-Degree Delphi yields complementary statements from different stakeholders, which would be lost in classical Delphi; while (2) the variation of statements within individual stakeholder groups is lower than within the total collective. A user study is performed that addresses five stakeholder groups (patients, relatives, medical doctors, nurses and software developers) on the topic of living will communication in an emergency context. Qualitative open questions are used in a Delphi round 0. Answer texts are coded by independent raters who carry out systematic bottom-up qualitative text analysis. Inter-rater reliability is calculated and the resulting codes are used to test the hypotheses. Qualitative results are transferred into quantitative questions and then surveyed in round 1. The study took place in Germany.

Results: About 25% of the invited experts (stakeholders) agreed to take part in the Delphi round 0 (three patients, two relatives, three medical doctors, two qualified nurses and three developers), forming a structured panel of the five stakeholder groups. Two raters created a bottom-up coding, and 238 thematic codes were identified by the qualitative text analysis. The inter-rater reliability showed that 44.95% of the codes were semantically similar and coded for the same parts of the raw textual replies. Based on a consented coding list, a quantitative online-questionnaire was developed and send to different stakeholder groups. With respect to the hypotheses, Delphi round 0 had the following results: (1) doctors had a completely different focus from all the other stakeholder groups on possible channels of communications with the patient; (2) the dispersion of codes within individual stakeholder groups and within the total collective - visualized by box plots - was approximately 28% higher in the total collective than in the sub-collectives, but without a marked effect size. With respect to the hypotheses, Delphi round 1 had the following results: different stakeholder groups had highly diverging opinions with respect to central questions on IT-development. For example, when asked to rate the importance of access control against high availability of data (likert scale, 1 meaning restrictive data access, 6 easy access to all data), patients (mean 4.862, Stdev +/- 1.866) and caregivers (mean 5.667, Stdev: +/- 0.816) highly favored data availability, while relatives would restrict data access (mean 2.778, stdev +/- 1.093). In comparison, the total group would not be representative of either of these individual stakeholder needs (mean 4.344, stdev +/- 1.870).

Conclusion: 360-Degree Delphi is feasible and allows different stakeholder groups within an expert panel to reach agreement individually. Thus, it generates a more detailed consensus which pays more tribute to individual stakeholders needs. This has the potential to improve the time to consensus as well as to produce a more representative and precise needs and requirements analysis. However, the method may create new challenges for the IT development process, which will have to deal with complementary or even contradictory statements from different stakeholder groups.

Keywords: 360-degree Delphi; Consensus; Delphi technique; Feedback; Group processes; Health information management; Medical informatics; Project planning; Qualitative research; Quality improvement; Stakeholder.

Fig. 1

Phases and corresponding survey rounds

Fig. 2

Structure of the 360-Degree Delphi (360°D) approach. Changes to the traditional Delphi technique and items that are now mandatory are marked with a green mark

Fig. 3

a Security vs accessibility, likert-scale answers of the stakeholder groups. b Restriction vs. accessibilty, likert-scale answers of the stakeholder groups. c Change vs. no data access, likert-scale answers of the stakeholder groups. d Privacy vs. accessibilty, likert-scale answers of the stakeholder groups. e Data recovery vs. no data access, likert-scale answers of the stakeholder groups

Fig. 4

Comparison of dispersion within total collective and individual stakeholder groups in Round 0