Predicting seizure outcome after epilepsy surgery: Do we need more complex models, larger samples, or better data?

Abstract

Objective: The accurate prediction of seizure freedom after epilepsy surgery remains challenging. We investigated if (1) training more complex models, (2) recruiting larger sample sizes, or (3) using data-driven selection of clinical predictors would improve our ability to predict postoperative seizure outcome using clinical features. We also conducted the first substantial external validation of a machine learning model trained to predict postoperative seizure outcome.

Methods: We performed a retrospective cohort study of 797 children who had undergone resective or disconnective epilepsy surgery at a tertiary center. We extracted patient information from medical records and trained three models-a logistic regression, a multilayer perceptron, and an XGBoost model-to predict 1-year postoperative seizure outcome on our data set. We evaluated the performance of a recently published XGBoost model on the same patients. We further investigated the impact of sample size on model performance, using learning curve analysis to estimate performance at samples up to N = 2000. Finally, we examined the impact of predictor selection on model performance.

Results: Our logistic regression achieved an accuracy of 72% (95% confidence interval [CI] = 68%-75%, area under the curve [AUC] = .72), whereas our multilayer perceptron and XGBoost both achieved accuracies of 71% (95% CI_MLP = 67%-74%, AUC_MLP = .70; 95% CI_{XGBoost own} = 68%-75%, AUC_{XGBoost own} = .70). There was no significant difference in performance between our three models (all p > .4) and they all performed better than the external XGBoost, which achieved an accuracy of 63% (95% CI = 59%-67%, AUC = .62; p_LR = .005, p_MLP = .01, p_{XGBoost own} = .01) on our data. All models showed improved performance with increasing sample size, but limited improvements beyond our current sample. The best model performance was achieved with data-driven feature selection.

Significance: We show that neither the deployment of complex machine learning models nor the assembly of thousands of patients alone is likely to generate significant improvements in our ability to predict postoperative seizure freedom. We instead propose that improved feature selection alongside collaboration, data standardization, and model sharing is required to advance the field.

Keywords: epilepsy surgery; machine learning; pediatric; prediction.

FIGURE 1

Study overview. We investigated the impact of model type, sample size, and feature selection on our ability to accurately predict postoperative seizure outcome.

FIGURE 2

Relationships between demographic, clinical, and surgical variables. Relationships are shown both before and after correction for multiple comparison using the Holm method. We have highlighted relationships with seizure outcome using a yellow box. ASM, antiseizure medication; Num. ASM pre‐op, number of antiseizure medications at time of preoperative evaluation; Num. ASM trialed, total number of different antiseizure medications trialed from epilepsy onset to preoperative evaluation.

FIGURE 3

Impact of model type and sample size on model performance. (A) Receiver‐ operating characteristic (ROC) curves showing model performances. There was no significant difference in performance between our LR (purple), MLP (pink), and XGBoost (teal) models. All of our models performed significantly better than the XGBoost model recently developed by Yossofzai et al. (light blue). (B) The effect of sample size on model performance (accuracy). There was an improvement in model performance with increasing sample size for our LR, MPL, and XGBoost models, but only up until a certain point. After this, the models showed only marginal gains in performance. Extrapolating performance for sample sizes up to N = 2000 did not predict substantial improvement in model performance for any of our models. AUC, area under the (ROC) curve; LR, logistic regression; MLP, multilayer perceptron; ROC, receiver‐operating characteristic.

FIGURE 4

Impact of feature selection on model performance. (A) Receiver‐operating characteristic (ROC) curves showing model performance for our LR models containing (1) only MRI diagnosis (red), (2) all predictors (orange), and (3) predictors identified through data‐driven feature selection (green). Data‐driven selection involved including only predictors that were significantly predictive of 1‐year postoperative seizure outcome as identified in univariable logistic regression analyses. Corresponding ROC curves showing model performances for our MLP and XGBoost models are displayed in Figures S2 and S3. (B) Effect of data‐driven feature selection on model performance (AUC). Variables found to be significantly predictive of seizure outcome from univariable logistic regression analyses were added to the LR, from most information to least informative according to their coefficients. Model performance was best when all significantly predictive features were included in the model. Adding the remaining predictors collected for the study, that is, those that were not significantly predictive of seizure outcome, worsened model performance (far right). Points circled in black represent mean AUC obtained across all 10 folds. Noncircled points represent the AUCs obtained from each of the individual 10 folds. ASM, antiseizure medication; AUC, area under the (ROC) curve; LR, logistic regression; NS. predictors, non‐significant predictors; Num. ASM trialed, total number of different antiseizure medication trialed from epilepsy onset to preoperative evaluation; Num. seiz. types, number of seizure types at time of preoperative evaluation; ROC, receiver‐operating characteristic; Spasms hist., history of spasms; Spasms pre‐op, spasms at time of preoperative evaluation.