Predicting seizure freedom after epilepsy surgery, a challenge in clinical practice

Abstract

Objective: The objective of this study was to compare the accuracy of clinical judgment in predicting seizure outcome after resective epilepsy surgery relative to two recently published statistical tools [the Epilepsy Surgery Nomogram (ESN) and the modified Seizure-Freedom score (m-SFS)].

Methods: Details of presurgical evaluations of 20 patients who underwent epilepsy surgery were presented to 20 epilepsy experts. The final surgical treatment was also disclosed. The clinicians were asked to predict the likelihood of a good outcome (Engel 1) at 2 and 5 years in each case. The ESN and the m-SFS predictions were calculated with the data provided to the clinicians. The discriminative ability of clinical judgment, ESN, and m-SFS was assessed by calculating a concordance index (C-index). Expert opinion, the m-SFS and the ESN performances were compared using a Receiver Operating Characteristic (ROC) curve analysis.

Results: The mean age at surgery was 29 years (standard deviation [SD] = 14); 40% were male; 70% were right-handed, and thirteen (65%) had an Engel outcome 1 at 2 and 5 years. The mean C-index for the mean physician's prediction was 0.478 with a variance of 0.012. The ESN had an area under the curve (AUC) of 0.528 and 0.533 for the 2-year and 5-year predictions in comparison with the clinicians' predictions that was 0.476, and 0.466, respectively. For the m-SFS, the AUC at 2 years and 5 years was 0.539 and 0.539, respectively. No statistical difference was noted between the ESN and the clinicians or between m-SFS and the ESN, but there is a moderate statistical difference favoring the m-SFS to the clinicians (p 0.0960 and 0.0514, for 2 and 5 years).

Significance: Clinical judgment was not superior to the ESN nor to the m-SFS. Together with the interphysician's prediction variability, our findings reinforce the need for better tools to predict postoperative outcomes.

Keywords: Clinical judgment; Epilepsy surgery; Presurgical evaluation; Seizure freedom.

Figure 1.

Scatter plot of each individual doctor’s predictions on all 20 patients. A. Plot for predicting at 2 years. B. Plot for predicting at 5 years. Each dot represents a predicted probability of seizure-freedom from the doctors in black circles, the ESN in red, and the m-SFS score in blue. The green boxes represent the patients who were actually seizure-free.

Figure 2.

Analysis of each doctor’s predictions on all 20 patients as well as the ESN and m-SFS predictions. A. Prediction at 2 years. B. Prediction 5 years. Each bar represents the C-index of each doctor’s prediction on all 20 patients, where the green bar is the ESN’s predictions and the blue bar is the SFS score predictions. The dotted horizontal line is the overall concordance index for all doctors. The mean concordance index for all doctors is 0.478 with a variance of 0.012.

Figure 3.

Analysis of each doctor’s predictions on all 20 patients as well as the ESN and m-SFS predictions, stratified by age groups. A. Plot for predicting 2 years. B. Plot for predicting 5 years. Each bar represents each doctor’s prediction on all 20 patients stratified by age groups 30–40, 41–50, 51–60, and >60, where the green bar is the ESN’s predictions and the blue bar is the m-SFS score predictions. No significant significant difference was noted among predictive models. The dotted horizontal line is the overall concordance index for all doctors.

Figure 4.

Analysis of each doctor’s predictions on all 20 patients as well as the ESNs predictions stratified by years out of training. A. Plot for predicting 2 years. B.Plot for predicting at 5 years. Each bar represents each doctor’s prediction on all 20 patients stratified by years out of training, where the green bar is the ESN’s predictions and the blue bar is the m-SFS score predictions. No significant significant difference was noted among predictive models. The dotted horizontal line is the overall concordance index for all doctors.

Figure 5.

Analysis of each doctor’s predictions on all 20 patients as well as the nomograms predictions stratified by number of surgeries per year. A. Plot for predicting 2 years. B. Plot for predicting at 5 years. Each bar represents each doctor’s prediction on all 20 patients stratified by number of surgeries per year, 10–24, 25–50, and >50, where the green bar is the ESN’s predictions and the blue bar is the m-SFS score predictions. No significant significant difference was noted among predictive models. The dotted horizontal line is the overall concordance index for all doctors.

Figure 6.

The ROC curve for 2 year predictions (A) and the curve for 5 year predictions (B). The ESN for 2 year and 5 year predictions has an AUC of 0.528 and 0.533, respectively, the doctors’ predictions have an AUC at 2 years and 5 years of 0.476, and 0.466, respectively, and the m-SFS scores have an AUC at 2 years and 5 years of 0.539 and 0.539, respectively. There is no statistical difference between the ESN and the doctors or between m-SFS and the ESN, but there is a moderate statistical difference between m-SFS and the doctors (p-value:0.0960 and 0.0514, for 2 and 5 years). Toward the higher end of Sensitivity and Specificity, both the ESN and the doctors are very similar and the SFS score seems to be very similar to the doctors.