Development of Expert-Level Automated Detection of Epileptiform Discharges During Electroencephalogram Interpretation

Abstract

Importance: Interictal epileptiform discharges (IEDs) in electroencephalograms (EEGs) are a biomarker of epilepsy, seizure risk, and clinical decline. However, there is a scarcity of experts qualified to interpret EEG results. Prior attempts to automate IED detection have been limited by small samples and have not demonstrated expert-level performance. There is a need for a validated automated method to detect IEDs with expert-level reliability.

Objective: To develop and validate a computer algorithm with the ability to identify IEDs as reliably as experts and classify an EEG recording as containing IEDs vs no IEDs.

Design, setting, and participants: A total of 9571 scalp EEG records with and without IEDs were used to train a deep neural network (SpikeNet) to perform IED detection. Independent training and testing data sets were generated from 13 262 IED candidates, independently annotated by 8 fellowship-trained clinical neurophysiologists, and 8520 EEG records containing no IEDs based on clinical EEG reports. Using the estimated spike probability, a classifier designating the whole EEG recording as positive or negative was also built.

Main outcomes and measures: SpikeNet accuracy, sensitivity, and specificity compared with fellowship-trained neurophysiology experts for identifying IEDs and classifying EEGs as positive or negative or negative for IEDs. Statistical performance was assessed via calibration error and area under the receiver operating characteristic curve (AUC). All performance statistics were estimated using 10-fold cross-validation.

Results: SpikeNet surpassed both expert interpretation and an industry standard commercial IED detector, based on calibration error (SpikeNet, 0.041; 95% CI, 0.033-0.049; vs industry standard, 0.066; 95% CI, 0.060-0.078; vs experts, mean, 0.183; range, 0.081-0.364) and binary classification performance based on AUC (SpikeNet, 0.980; 95% CI, 0.977-0.984; vs industry standard, 0.882; 95% CI, 0.872-0.893). Whole EEG classification had a mean calibration error of 0.126 (range, 0.109-0.1444) vs experts (mean, 0.197; range, 0.099-0.372) and AUC of 0.847 (95% CI, 0.830-0.865).

Conclusions and relevance: In this study, SpikeNet automatically detected IEDs and classified whole EEGs as IED-positive or IED-negative. This may be the first time an algorithm has been shown to exceed expert performance for IED detection in a representative sample of EEGs and may thus be a valuable tool for expedited review of EEGs.

Figure 1.. Example Electroencephalographic Segments of Interictal Epileptiform Discharges Identified at Each Expert Agreement Level

Standard banana, bipolar montage 1-second displays with expert agreement ratios: number of experts identifying that segment as containing a spike divided by total number of experts who performed annotations displayed across the top (eg, 0/8 indicates none of the 8 experts).

Figure 2.. SpikeNet Performance Evaluation

A, The receiver operating characteristic curves (ROCs) and 95% CIs for SpikeNet (area under the curve [AUC], 0.980; 95% CI, 0.977-0.984) and the industry standard (Persyst 13) (AUC, 0.882; 95% CI, 0.872-0.893) when identifying interictal epileptiform discharges (IEDs) vs non-IEDs, where spikes were confirmed by more than 6 of 8 experts vs fewer than 2 of 8 experts as the definition of definite IED vs definite non-IED. B, Calibration curves for SpikeNet, Persyst 13, and human experts for IED detection compared with a perfect calibration line (dotted). C, The ROC curve and 95% CI for SpikeNet’s whole electroencephalogram (EEG) classification (AUC, 0.847; 95% CI, 0.830-0.865). D, The calibration curves for SpikeNet vs human experts for whole EEG classification. Shaded areas represent the 95% CIs.