Quality assessment of expert answers to lay questions about cystic fibrosis from various language zones in Europe: the ECORN-CF project

Abstract

Background: The European Centres of Reference Network for Cystic Fibrosis (ECORN-CF) established an Internet forum which provides the opportunity for CF patients and other interested people to ask experts questions about CF in their mother language. The objectives of this study were to: 1) develop a detailed quality assessment tool to analyze quality of expert answers, 2) evaluate the intra- and inter-rater agreement of this tool, and 3) explore changes in the quality of expert answers over the time frame of the project.

Methods: The quality assessment tool was developed by an expert panel. Five experts within the ECORN-CF project used the quality assessment tool to analyze the quality of 108 expert answers published on ECORN-CF from six language zones. 25 expert answers were scored at two time points, one year apart. Quality of answers was also assessed at an early and later period of the project. Individual rater scores and group mean scores were analyzed for each expert answer.

Results: A scoring system and training manual were developed analyzing two quality categories of answers: content and formal quality. For content quality, the grades based on group mean scores for all raters showed substantial agreement between two time points, however this was not the case for the grades based on individual rater scores. For formal quality the grades based on group mean scores showed only slight agreement between two time points and there was also poor agreement between time points for the individual grades. The inter-rater agreement for content quality was fair (mean kappa value 0.232 ± 0.036, p < 0.001) while only slight agreement was observed for the grades of the formal quality (mean kappa value 0.105 ± 0.024, p < 0.001). The quality of expert answers was rated high (four language zones) or satisfactory (two language zones) and did not change over time.

Conclusions: The quality assessment tool described in this study was feasible and reliable when content quality was assessed by a group of raters. Within ECORN-CF, the tool will help ensure that CF patients all over Europe have equal possibility of access to high quality expert advice on their illness.

Figure 1

Content quality of answers from the Dutch language zone during the early and later period of the ECORN-CF system. Ten expert answers from the early period (1-10) and ten expert answers from the later period (11-20) were assessed. Each column represents the mean of the four scores given to a single answer (one additional rater scored only the formal quality). The squares represent the highest and lowest scores given to that answer. The blue columns at the end of each group of ten expert answers represent the mean of the scores for all answers of the early period and the later period, respectively; they did not differ between early and later period (p = 0.095). Dark grey columns represent answers with a mean score of "good" quality.

Figure 2

Formal quality of answers from the Dutch language zone during the early and later period of the ECORN-CF system. Ten expert answers from the early period (1-10) and ten expert answers from the later period (11-20) were assessed. Each column represents the mean of the five scores given to a single answer. The squares represent the highest and lowest scores given to that answer. The blue columns at the end of each group of ten expert answers represent the mean of the scores for all answers of the early period and the later period, respectively; they did not differ between early and later period (p = 0.265). Dark grey columns represent answers with a mean score of "good" quality, light grey columns those of "satisfactory" quality.

Figure 3

Content quality of answers in all six language zones during the early and later period of the ECORN-CF system. Each column represents the mean of all scores for the expert answers from the respective period marked as "early" or "later" and the respective language zone. The squares represent the highest and lowest mean score given to a single answer. The blue columns marked as "all" represent the mean of the scores for all 50 expert answers from all language zones of the early period and for all 58 expert answers from all language zones of the later period, respectively. Dark grey columns represent answers of "good" quality, light grey columns those of "satisfactory" quality. B/NL = Belgium/Netherlands, CZ = Czech Republic, GER = Germany, LIT = Lithuania, ROM = Romania, UK = United Kingdom.

Figure 4

Formal quality of answers in all six language zones during the early and later period of the ECORN-CF system. Each column represents the mean of all scores for the expert answers from the respective period marked as "early" or "later" and the respective language zone. The squares represent the highest and lowest mean score given to a single answer. The blue columns marked as "all" represent the mean of the scores for all 50 expert answers from all language zones of the early period and for all 58 expert answers from all language zones of the later period, respectively. The squares represent the highest and lowest mean score given to a single answer. Dark grey columns represent answers of "good" quality, light grey columns those of "satisfactory" quality. B/NL = Belgium/Netherlands, CZ = Czech Republic, GER = Germany, LIT = Lithuania, ROM = Romania, UK = United Kingdom.