Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms

Abstract

Importance: The validity of using electroencephalograms (EEGs) to diagnose epilepsy requires reliable detection of interictal epileptiform discharges (IEDs). Prior interrater reliability (IRR) studies are limited by small samples and selection bias.

Objective: To assess the reliability of experts in detecting IEDs in routine EEGs.

Design, setting, and participants: This prospective analysis conducted in 2 phases included as participants physicians with at least 1 year of subspecialty training in clinical neurophysiology. In phase 1, 9 experts independently identified candidate IEDs in 991 EEGs (1 expert per EEG) reported in the medical record to contain at least 1 IED, yielding 87 636 candidate IEDs. In phase 2, the candidate IEDs were clustered into groups with distinct morphological features, yielding 12 602 clusters, and a representative candidate IED was selected from each cluster. We added 660 waveforms (11 random samples each from 60 randomly selected EEGs reported as being free of IEDs) as negative controls. Eight experts independently scored all 13 262 candidates as IEDs or non-IEDs. The 1051 EEGs in the study were recorded at the Massachusetts General Hospital between 2012 and 2016.

Main outcomes and measures: Primary outcome measures were percentage of agreement (PA) and beyond-chance agreement (Gwet κ) for individual IEDs (IED-wise IRR) and for whether an EEG contained any IEDs (EEG-wise IRR). Secondary outcomes were the correlations between numbers of IEDs marked by experts across cases, calibration of expert scoring to group consensus, and receiver operating characteristic analysis of how well multivariate logistic regression models may account for differences in the IED scoring behavior between experts.

Results: Among the 1051 EEGs assessed in the study, 540 (51.4%) were those of females and 511 (48.6%) were those of males. In phase 1, 9 experts each marked potential IEDs in a median of 65 (interquartile range [IQR], 28-332) EEGs. The total number of IED candidates marked was 87 636. Expert IRR for the 13 262 individually annotated IED candidates was fair, with the mean PA being 72.4% (95% CI, 67.0%-77.8%) and mean κ being 48.7% (95% CI, 37.3%-60.1%). The EEG-wise IRR was substantial, with the mean PA being 80.9% (95% CI, 76.2%-85.7%) and mean κ being 69.4% (95% CI, 60.3%-78.5%). A statistical model based on waveform morphological features, when provided with individualized thresholds, explained the median binary scores of all experts with a high degree of accuracy of 80% (range, 73%-88%).

Conclusions and relevance: This study's findings suggest that experts can identify whether EEGs contain IEDs with substantial reliability. Lower reliability regarding individual IEDs may be largely explained by various experts applying different thresholds to a common underlying statistical model.

Figure 1.. Flow Diagram of the Study

EEG indicates electroencephalogram; IEDs, interictal epileptiform discharges.

Figure 2.. Interrater Reliability (IRR) for Interictal Epileptiform Discharges (IEDs) at the Level of Individual IEDs and Entire Electroencephalograms (EEGs)

A, The 8 rows each contain 10 randomly selected samples scored by 8 experts as being IEDs. Qualitatively, expert IRR increases in proportion to the degree that candidate waves exhibit the morphological features of IEDs as defined by IFSECN (International Federation of Societies for EEG and Clinical Neurophysiology) criteria. B, The IED-wise percentage of agreement between pairs of experts. C, The EEG-wise percentage of agreement among pairs of experts. D, The EEGs are arranged in order of the mean number of IEDs marked by all experts, and experts are arranged from top to bottom in order of the total number of IEDs they marked across all EEGs. E and F, The IEDs were binned according to the number of votes (0 through 8) that they received (termed the reference standard). The 8 calibration curves, 1 for each expert, indicate the probability of that expert marking events within a given bin as IEDs. These curves allow assessment of the variation among experts relative to the group consensus.

Figure 3.. Morphological Characteristics of the Interictal Epileptiform Discharges (IEDs)

A, For each candidate IED, 5 fiducial points are identified (triangles) corresponding to the IED peak, troughs preceding and following the peak, peak of the after-going slow wave, and trough following the slow-wave peak. These feature values are used to construct a single multivariate logistic regression (MLR) model to evaluate binary IED scores for all experts combined (combined model) and individualized MLR models to evaluate the IED scores of individual experts (individualized models). B, Receiver operating characteristic curve for the MLR model fit to the scores of all experts (universal model). The operating point (false-positive rate = 1 − specificity and sensitivity) of each expert, corresponding to the threshold in the combined model that best evaluates that expert’s binary scores, is indicated by a solid circle. The number by each circle is the threshold that best evaluates that expert’s binary scores. A indicates the area under the IED curve; A_p, area under the peak; A_s, area under the slow wave; AUC, area under the curve; D, duration of the IED candidate wave; D_p, duration of the peak; D_pf, duration of the falling half-wave of peak; D_pr, duration of the rising half-wave of peak; D_s, duration of the slow wave; D_sf, duration of the falling half-wave of the slow wave; D_sr, duration of the rising half-wave of the slow wave; Ref, reference; S_pf, slope of the falling half-wave of peak; S_sf, slope of the falling half-wave of the slow wave; S_pr, slope of the rising half-wave of peak; S_sr, slope of the rising half-wave of the slow wave; V_p0, peak voltage with respect to baseline 0 defined by V = 0; V_p1, peak voltage with respect to baseline 1 defined by V_on; V_p2, peak voltage with respect to baseline 2 defined by trough V_tr; V_s0, slow-wave peak voltage with respect to baseline 0 defined by V = 0; V_s1, slow-wave peak voltage with respect to baseline 1 defined by onset V_on; V_s2, slow-wave peak voltage with respect to baseline 2 defined by trough V_tr; V_t0, trough voltage with respect to baseline 0 defined by V = 0; and V_t1, trough voltage with respect to baseline 1 defined by onset V_on.