Crowdsourcing seizure detection: algorithm development and validation on human implanted device recordings

Abstract

There exist significant clinical and basic research needs for accurate, automated seizure detection algorithms. These algorithms have translational potential in responsive neurostimulation devices and in automatic parsing of continuous intracranial electroencephalography data. An important barrier to developing accurate, validated algorithms for seizure detection is limited access to high-quality, expertly annotated seizure data from prolonged recordings. To overcome this, we hosted a kaggle.com competition to crowdsource the development of seizure detection algorithms using intracranial electroencephalography from canines and humans with epilepsy. The top three performing algorithms from the contest were then validated on out-of-sample patient data including standard clinical data and continuous ambulatory human data obtained over several years using the implantable NeuroVista seizure advisory system. Two hundred teams of data scientists from all over the world participated in the kaggle.com competition. The top performing teams submitted highly accurate algorithms with consistent performance in the out-of-sample validation study. The performance of these seizure detection algorithms, achieved using freely available code and data, sets a new reproducible benchmark for personalized seizure detection. We have also shared a 'plug and play' pipeline to allow other researchers to easily use these algorithms on their own datasets. The success of this competition demonstrates how sharing code and high quality data results in the creation of powerful translational tools with significant potential to impact patient care.

Keywords: crowdsourcing; epilepsy; experimental models; intracranial EEG; seizure detection.

Figure 1

Representative EEG data. (A) MRI imaging of patient with implanted NeuroVista SAS device. This device was used for collection of canine data in the competition and human data in the validation study. (B) Sample EEG recording of a seizure. Vertical lines represent 1-s intervals. Boundaries of seizure and early seizure periods are marked.

Figure 2

Top algorithm performance over time. Leading score over the course of the kaggle.com competition on public (blue) and private (red) leader boards. The top score in the validation study is represented by the dashed grey line.

Figure 3

Validation study performance. Performance of (left to right bars) Algorithm 1 (blue), Algorithm 2 (red), Algorithm 3 (green), and the ensemble Algorithm (grey) on each cohort in the validation study. Each point represents performance on an individual subject.

Figure 4

Validation study ROC curves. ROC curves for (A) seizure classification in the HUP and Mayo cohorts, (B) early seizure classification in the HUP and Mayo cohorts, (C) seizure classification in the NeuroVista cohort, (D) early seizure classification in the NeuroVista cohort.

Figure 5

Representative seizure detection in the low-false-positive limit. Each EEG trace shows a single representative channel signal from a different seizure with 40 s of preictal recording. The seizure EEC is denoted by the dashed line. Seizures shown were all derived from Patient NV1. Areas highlighted in red were classified as seizure by Algorithm 1.