Interrater agreement for Critical Care EEG Terminology

Abstract

Objective: The interpretation of critical care electroencephalography (EEG) studies is challenging because of the presence of many periodic and rhythmic patterns of uncertain clinical significance. Defining the clinical significance of these patterns requires standardized terminology with high interrater agreement (IRA). We sought to evaluate IRA for the final, published American Clinical Neurophysiology Society (ACNS)-approved version of the critical care EEG terminology (2012 version). Our evaluation included terms not assessed previously and incorporated raters with a broad range of EEG reading experience.

Methods: After reviewing a set of training slides, 49 readers independently completed a Web-based test consisting of 11 identical questions for each of 37 EEG samples (407 questions). Questions assessed whether a pattern was an electrographic seizure; pattern location (main term 1), pattern type (main term 2); and presence and classification of eight other key features ("plus" modifiers, sharpness, absolute and relative amplitude, frequency, number of phases, fluctuation/evolution, and the presence of "triphasic" morphology).

Results: IRA statistics (κ values) were almost perfect (90-100%) for seizures, main terms 1 and 2, the +S modifier (superimposed spikes/sharp waves or sharply contoured rhythmic delta activity), sharpness, absolute amplitude, frequency, and number of phases. Agreement was substantial for the +F (superimposed fast activity) and +R (superimposed rhythmic delta activity) modifiers (66% and 67%, respectively), moderate for triphasic morphology (58%), and fair for evolution (21%).

Significance: IRA for most terms in the ACNS critical care EEG terminology is high. These terms are suitable for multicenter research on the clinical significance of critical care EEG patterns. A PowerPoint slide summarizing this article is available for download in the Supporting Information section http://dx.doi.org/10.1111/epi.12653/supinfo.

Keywords: Continuous EEG monitoring; Critical care; EEG terminology; GPEDs; Intensive care; Interrater agreement; PLEDs; Periodic patterns; Rhythmic patterns.

Figure 1

(A) Detailed individual scores. The heatmap shows scores, coded by color according to the colorbar on the right, for all 49 individuals and for each of 15 different ACNS Critical Care EEG terminology concepts. Scores for each concept are calculated as % of answers in agreement with answers determined by consensus of an expert panel. Respondents are rank-ordered according to overall score. (B). Overall individual scores. Scores from (A) are summarized as a single score obtained in three different ways: By averaging the scores over all 15 concepts, giving equal weight to each (red bars); by averaging with terms weighted by the percentage agreement (orange bars); and by averaging with terms weighted by the IRA value (κ) for each concept (white bars). Sz, seizure; M1, main term 1; M2, main term 2; +F, superimposed fast activity; +R, superimposed rhythmic delta activity; +S, superimposed sharp waves or spikes, or sharply contoured activity; A+, any “plus” modifier (and of +F, +R, or +S vs. No+); C+, specific combination of “plus” modifiers (F, R, S, FR, FS, or No +); Sh, sharpness; AA, absolute amplitude; RA, relative amplitude; Fr, frequency; Ph, number of phases; Ev, evolution; TP, triphasic morphology. Epilepsia © ILAE

Figure 2

Kappa (κ) values in relation to percent agreement. Horizontal bars show the percent agreement (PA, red bars + orange bars), relative to the maximal possible percent agreement (100%, ends of white bars). The lengths of red bars show the percent agreement estimated to be attributable to chance, PC, used in estimating the degree of IRA, κ. The total percentage of possible agreement beyond each red bar, 100-PC (orange bar + white bar) is considered the potential degree of agreement achievable “beyond chance,” whereas the beyond-chance agreement actually achieved is PA–PC (the length of the orange bars). Mathematically, the chance-corrected IRA, κ, is the percentage of this possible beyond-chance agreement that is actually achieved, that is, $\kappa =\frac{\text{PA}-\text{PC}}{100-\text{PC}}$. Graphically, this mathematical definition of κ is represented as the fraction of the distance between 100% and the end of the red bar that is taken up by the orange bar. Abbreviations are the same as in Figure 1. Epilepsia © ILAE

Figure 3

Confusion matrices. Color-coded confusion matrices for 10 ACNS standardized critical care EEG terminology items are displayed as heat map (scale bar on the right). The available choices for each concept are shown along the vertical and horizontal axes. Heat map intensities indicate the percentage of respondents who chose each option along the horizontal axis when the correct response (determined by panel-of-experts consensus) was the one on the vertical axis. The percentage of respondents choosing each available option is displayed on a color scale along the horizontal (column) axes. A perfect result would produce a diagonal white line from upper left to lower right, with all squares not on or adjacent to that line being black. The majority of respondents selected the correct term on average in all cases shown. It is notable that the minority of incorrect responses was generally “close” to the correct response (near the diagonal line). Abbreviations are as in Figure 1. Epilepsia © ILAE

Figure 4

Effects of EEG experience on individual performances. Scores (% correct, relative to expert panel consensus answers) were plotted for all raters within five groups of self-reported EEG reading experience: 0–2, 2–5, 5–10, 10–15, and >15 years. Linear regression lines (red dashed lines, of the form y = slope x + b), were fitted to each set data, with x values equal to 1, 2, 3, 4, or 5 for the different levels of experience. The slopes and associated p-values are shown on each graph. Epilepsia © ILAE