Large-Scale Automated Sleep Staging

Abstract

Study objectives: Automated sleep staging has been previously limited by a combination of clinical and physiological heterogeneity. Both factors are in principle addressable with large data sets that enable robust calibration. However, the impact of sample size remains uncertain. The objectives are to investigate the extent to which machine learning methods can approximate the performance of human scorers when supplied with sufficient training cases and to investigate how staging performance depends on the number of training patients, contextual information, model complexity, and imbalance between sleep stage proportions.

Methods: A total of 102 features were extracted from six electroencephalography (EEG) channels in routine polysomnography. Two thousand nights were partitioned into equal (n = 1000) training and testing sets for validation. We used epoch-by-epoch Cohen's kappa statistics to measure the agreement between classifier output and human scorer according to American Academy of Sleep Medicine scoring criteria.

Results: Epoch-by-epoch Cohen's kappa improved with increasing training EEG recordings until saturation occurred (n = ~300). The kappa value was further improved by accounting for contextual (temporal) information, increasing model complexity, and adjusting the model training procedure to account for the imbalance of stage proportions. The final kappa on the testing set was 0.68. Testing on more EEG recordings leads to kappa estimates with lower variance.

Conclusion: Training with a large data set enables automated sleep staging that compares favorably with human scorers. Because testing was performed on a large and heterogeneous data set, the performance estimate has low variance and is likely to generalize broadly.

Keywords: EEG; big data; machine learning; sleep stages.

Figure 1

(A) Cohen’s kappa for different numbers of training patients and the fixed 1000 testing patients. The testing Cohen’s kappa was computed using the confusion matrix of all epochs from all 1000 testing patients. The mean and standard deviation of the kappa values from five repetitions are displayed. The ** or * markers indicate adjacent cases that exhibit a statistically significant improvement in the kappa statistic, based on Mann-Whitney U test (<0.01 or <0.05, respectively). (B) Cohen’s kappa values after HMM smoothing, which will be described in the next subsection. HMM = hidden Markov model.

Figure 2

The confusion matrix of the fixed 1000 testing patients when trained with (A) 10 patients and 2000 hidden nodes; (B) 300 patients and 2000 hidden nodes; (C) 1000 patients and 2000 hidden nodes; (D) 1000 patients and 20,000 hidden nodes; (E) 1000 patients, 2000 hidden nodes and weighted training samples; and (F) 1000 patients, 20,000 hidden nodes, weighted training samples and smoothing. For each confusion matrix, the repetition with kappa value closest to the mean kappa value over five repetitions is shown. Values are given as percentages, which sum to 100 across rows (human scoring). The color is white for 100% and black for 0%.

Figure 3

Hypnograms before and after HMM smoothing for the patient with largest post-smoothing improvement in the Cohen’s kappa. HMM = hidden Markov model.

Figure 4

Hypnograms before and after HMM smoothing for the patient with largest post-smoothing decline in the Cohen’s kappa. HMM = hidden Markov model.

Figure 5

(A) Training and testing performance with different values of the regularization parameter C; (B) with different numbers of hidden nodes L. The mean and standard deviation of kappa values over five repetitions are shown in both panels

Figure 6

The histogram of the Cohen’s kappa of each testing patient. The dashed line at 0.684 indicates the overall Cohen’s kappa for all epochs pooled from all testing patients.

Figure 7

The testing Cohen’s kappa for different scorers. The numbers in the x-axis labels are number of patients scored by the scorer. Kruskal-Wallis H test followed by post hoc Dunn’s test suggests that the kappa values for S2 is higher than S1 and S3. Other scorers have similar kappa values. ** p-value < .01.

Figure 8

The precision of Cohen’s kappa statistics for subsets of 1000 testing patients with different sizes. The red dashed line is the training Cohen’s kappa. The blue solid line in the middle of the shading is the mean value of testing Cohen’s kappa from four randomly selected nonoverlapping patient subsets. Shading indicates the area of ±standard deviation.