High inter-reviewer variability of spike detection on intracranial EEG addressed by an automated multi-channel algorithm

Abstract

Objective: The goal of this study was to determine the consistency of human reviewer spike detection and then develop a computer algorithm to make the intracranial spike detection process more objective and reliable.

Methods: Three human reviewers marked interictal spikes on samples of intracranial EEGs from 10 patients. The sensitivity, precision and agreement in channel ranking by activity were calculated between reviewers. A computer algorithm was developed to parallel the way human reviewers detect spikes by first identifying all potential spikes on each channel using frequency filtering and then block scaling all channels at the same time in order to exclude potential spikes that fall below an amplitude and slope threshold. Its performance was compared to the human reviewers on the same set of patients.

Results: Human reviewers showed surprisingly poor inter-reviewer agreement, but did broadly agree on the ranking of channels for spike activity. The computer algorithm performed as well as the human reviewers and did especially well at ranking channels from highest to lowest spike frequency.

Conclusions: Our algorithm showed good agreement with the different human reviewers, even though they demonstrated different criteria for what constitutes a 'spike' and performed especially well at the clinically important task of ranking channels by spike activity.

Significance: An automated, objective method to detect interictal spikes on intracranial recordings will improve both research and the surgical management of epilepsy patients.

Figure 1. Spike detection using filtering and block scaling balances differences in human reviewers

(A) Flowchart overview of the detection algorithm. (B) The algorithm's main initial screening step involves bandpass filtering the data from 20-50Hz (bottom), which makes interictal spikes (blue arrows) stand out from background compared to more standard viewing filters (top). This method accentuates both large, obvious spikes (left) as well as those that may have otherwise been lost in larger slow wave background activity (right). (C) Scaling all channels together as a block preserves differences in spike detection between high spike frequency and low spike frequency channels. A set of 5 channels is shown from the original ECoG, after marking individual channels independently (channel scaling), and after block scaling. * shows the marked spikes.

Figure 2. Human reviewer and algorithm agreement across a range of patients

Patients chosen for this study displayed a wide range of interictal spike types as well as background activities on intracranial electrocorticography. Spike detections by the three human reviewers and the spike detection algorithm are shown for 4 of the 10 patients representing different patterns. The boxes above the spikes show where at least one reviewer or the automated spike detection algorithm marked a spike, denoted by the presence of a colored square within the box. Agreement was best when interictal spikes were very large and stood out strongly from background (Patients 7 and 10) and poorest when spikes are small and characterized primarily by the subsequent slow wave (Patient 3).

Figure 3. Heatmaps of interictal spike frequency

Heatmaps of interictal spike frequency superimposed on a patient's 3-dimensional brain rendering show that the three human reviewers rank channels similarly, but not identically (top). The spike detection algorithm, however, balances reviewer discrepancies and produces a similar pattern to the average of all three human reviewers (bottom).