Languages for different health information readers: multitrait-multimethod content analysis of Cochrane systematic reviews textual summary formats

Abstract

Background: Although subjective expressions and linguistic fluency have been shown as important factors in processing and interpreting textual facts, analyses of these traits in textual health information for different audiences are lacking. We analyzed the readability and linguistic psychological and emotional characteristics of different textual summary formats of Cochrane systematic reviews.

Methods: We performed a multitrait-multimethod cross-sectional study of Press releases available at Cochrane web site (n = 162) and corresponding Scientific abstracts (n = 158), Cochrane Clinical Answers (n = 35) and Plain language summaries in English (n = 156), French (n = 101), German (n = 41) and Croatian (n = 156). We used SMOG index to assess text readability of all text formats, and natural language processing tools (IBM Watson Tone Analyzer, Stanford NLP Sentiment Analysis and Linguistic Inquiry and Word Count) to examine the affective states and subjective information in texts of Scientific abstracts, Plain language summaries and Press releases.

Results: All text formats had low readability, with SMOG index ranging from a median of 15.6 (95% confidence interval (CI) 15.3-15.9) for Scientific abstracts to 14.7 (95% CI 14.4-15.0) for Plain language summaries. In all text formats, "Sadness" was the most dominantly perceived emotional tone and the style of writing was perceived as "Analytical" and "Tentative". At the psychological level, all text formats exhibited the predominant "Openness" tone, and Press releases scored higher on the scales of "Conscientiousness", "Agreeableness" and "Emotional range". Press releases had significantly higher scores than Scientific abstracts and Plain language summaries on the dimensions of "Clout", and "Emotional tone".

Conclusions: Although the readability of Plain language summaries was higher than that of text formats targeting more expert audiences, the required literacy was much higher than the recommended US 6th grade level. The language of Press releases was generally more engaging than that of Scientific abstracts and Plain language summaries, which are written by the authors of systematic reviews. Preparation of textual summaries about health evidence for different audiences should take into account readers' subjective experiences to encourage cognitive processing and reaction to the provided information.

Keywords: Comprehension; Consumer health information; Health literacy; Natural language processing.

Fig. 1

Cochrane systematic review summary text formats included in the analysis. The starting cohort for sample formation was the collection of Cochrane Press releases in February 2016

Fig. 2

SMOG (“simple measure of gobbledygook”) readability index (number of years of education needed for a person to understand a written text) of Cochrane Scientific abstracts (SA) (n = 158); Press releases (PR) (n = 162); Plain language summaries (PLS) in English (n = 156), German (n = 41), French (n = 101) and Croatian (n = 156); and Cochrane Clinical Answers (CCA) (n = 35). Error bars are 95% confidence intervals. The full analysis of readability dataset is available Additional file 1: Table S1 and S2. SMOG index for Croatian language was calculated according to the formula adapted to Croatian [17]. Statistically significant differences (Kruskal Wallis test and post-hoc Conover Iman test): for summary formats in English – PR vs SA and PLS, CCA vs. PLS, SA vs. PR and PLS, PLS vs. all other formats; for PLS translations – all comparisons were significant

Fig. 3

Emotional and writing tone analysis of Cochrane Scientific abstracts (SA) (n = 158), Press releases (PR) (n = 162), and English Plain language summaries (PLS) (n = 156). The results of IBM Watson Tone Analyzer are expressed as the probability of the output variable to be present in the text. Error bars are 95% confidence intervals. The full analysis of the dataset is available in Additional file 1: Table S3. Tones with scores less than 0.5 are unlikely to be perceived in the content; scores over 0.75 mean that the measured tone will be perceived as dominant in the text [20]. Statistically significant differences (one way ANOVA and Tukey post-hoc test): “Anger”, “Fear” and “Analytical” – SA vs. PR and PLS, “Tentative” – SA vs. PLS

Fig. 4

Personality tone analysis of Cochrane Scientific abstracts (SA) (n = 158), Press releases (PR) (n = 162), and English Plain language summaries (PLS) (n = 156). The results of IBM Watson Tone Analyzer are expressed as the probability of the output variable to be present in the text. Error bars are 95% confidence intervals. The full analysis of the dataset is available in Additional file 1: Table S3. Tones with scores less than 0.5 are unlikely to be perceived in the content; scores over 0.75 mean that the measured tone will be perceived as dominant in the text [20]. Statistically significant differences (one way ANOVA and Tukey post-hoc test): “Openness” – SA vs. PR and PLS, “Conscientiousness”, “Agreeableness” and “Emotional range” – PR vs. SA and PLS

Fig. 5

Summary variables of language style of Cochrane Scientific abstracts (SA) (n = 158), Press releases (PR) (n = 162), and English Plain language summaries (PLS) (n = 156) according to LIWC analyser [21]. Results represent standardized scores converted to percentiles. Error bars are 95% confidence intervals. The full analysis of the dataset is available in Additional file 1: Table S4 and S5. Statistically significant differences (one way ANOVA and Tukey post-hoc test): “Analytical thinking” – all groups different, “Authenticity” – PLS vs. SA and PR, “Clout” and “Emotional tone” – PR vs. SA and PLS