Inter-rater reliability of the Full Outline of UnResponsiveness score and the Glasgow Coma Scale in critically ill patients: a prospective observational study

Abstract

Introduction: The Glasgow Coma Scale (GCS) is the most widely used scoring system for comatose patients in intensive care. Limitations of the GCS include the impossibility to assess the verbal score in intubated or aphasic patients, and an inconsistent inter-rater reliability. The FOUR (Full Outline of UnResponsiveness) score, a new coma scale not reliant on verbal response, was recently proposed. The aim of the present study was to compare the inter-rater reliability of the GCS and the FOUR score among unselected patients in general critical care. A further aim was to compare the inter-rater reliability of neurologists with that of intensive care unit (ICU) staff.

Methods: In this prospective observational study, scoring of GCS and FOUR score was performed by neurologists and ICU staff on 267 consecutive patients admitted to intensive care.

Results: In a total of 437 pair wise ratings the exact inter-rater agreement for the GCS was 71%, and for the FOUR score 82% (P = 0.0016); the inter-rater agreement within a range of +/- 1 score point for the GCS was 90%, and for the FOUR score 92% (P = ns.). The exact inter-rater agreement among neurologists was superior to that among ICU staff for the FOUR score (87% vs. 79%, P = 0.04) but not for the GCS (73% vs. 73%). Neurologists and ICU staff did not significantly differ in the inter-rater agreement within a range of +/- 1 score point for both GCS (88% vs. 93%) and the FOUR score (91% vs. 88%).

Conclusions: The FOUR score performed better than the GCS for exact inter-rater agreement, but not for the clinically more relevant agreement within the range of +/- 1 score point. Though neurologists outperformed ICU staff with regard to exact inter-rater agreement, the inter-rater agreement of ICU staff within the clinically more relevant range of +/- 1 score point equalled that of the neurologists. The small advantage in inter-rater reliability of the FOUR score is most likely insufficient to replace the GCS, a score with a long tradition in intensive care.

Figure 1

Frequency distribution of GCS and FOUR scores. Frequency distribution of 814 rated Glasgow Coma Scale (GCS) scores (top panel) and 814 rated Full Outline of UnResponsiveness (FOUR) scores (bottom panel).

Figure 2

Inter-rater agreement of GCS and FOUR scores. Scatterplots of the pair-wise ratings of neurologists (top panels), ICU staff (middle panels), and neurologist-ICU staff (bottom panels) for the Glasgow Coma Scale (GCS; left side panels) and Full Outline of UnResponsiveness (FOUR) score (right side panels).

Figure 3

Disagreement rates for GCS and FOUR scores. Disagreements of more than one score point in pair-wise ratings of the Glasgow Coma Scale (GCS) score (top panel) and the Full Outline of UnResponsiveness (FOUR) score (bottom panel) respectively. Scores are divided into quartiles. As a substantial proportion of ratings were at the maximum of the each scale (i.e. GCS 15, FOUR 16), the maximum category is shown separately in addition to the quartiles. Disagreements are expressed as a percentage of the total number of ratings in a given quartile of the GCS score and FOUR score, respectively. White bars = disagreements between the neurologists; black bars = disagreements between the neurologists and ICU staff; grey bars = disagreements between ICU staff. For both scores, disagreements were significantly (P < 0.001) less frequent in the maximum category (i.e. GCS 15, FOUR 16) than in all other categories. * For the lowest quartile of the FOUR score, the disagreement between neurologist and ICU staff (P = 0.034) and between ICU staff and ICU staff (P = 0.045) was significantly greater than that between the neurologists.

Figure 4

Predictive value for 28-day mortality. Receiver operating characteristic curve for the predictive value of Glasgow Coma Scale (GCS), Full Outline of UnResponsiveness (FOUR) score, and acute physiology and chronic health evaluation (APACHE) II score on 28-day mortality. There was no statistically significant difference between the areas under the curve of the three scores.