Multiple classifier systems for automatic sleep scoring in mice

Abstract

Background: Electroencephalogram (EEG) and electromyogram (EMG) recordings are often used in rodents to study sleep architecture and sleep-associated neural activity. These recordings must be scored to designate what sleep/wake state the animal is in at each time point. Manual sleep-scoring is very time-consuming, so machine-learning classifier algorithms have been used to automate scoring.

New method: Instead of using single classifiers, we implement a multiple classifier system. The multiple classifier is built from six base classifiers: decision tree, k-nearest neighbors, naïve Bayes, support vector machine, neural net, and linear discriminant analysis. Decision tree and k-nearest neighbors were improved into ensemble classifiers by using bagging and random subspace. Confidence scores from each classifier were combined to determine the final classification. Ambiguous epochs can be rejected and left for a human to classify.

Results: Support vector machine was the most accurate base classifier, and had error rate of 0.054. The multiple classifier system reduced the error rate to 0.049, which was not significantly different from a second human scorer. When 10% of epochs were rejected, the remaining epochs' error rate dropped to 0.018.

Comparison with existing method(s): Compared with the most accurate single classifier (support vector machine), the multiple classifier reduced errors by 9.4%. The multiple classifier surpassed the accuracy of a second human scorer after rejecting only 2% of epochs.

Conclusions: Multiple classifier systems are an effective way to increase automated sleep scoring accuracy. Improvements in autoscoring will allow sleep researchers to increase sample sizes and recording lengths, opening new experimental possibilities.

Keywords: Autoscoring; Electroencephalogram; Machine learning; Mouse; Multiple classifier system; Sleep; Sleep scoring.

Figure 1

Overall organization of the multiple classifier system. 21 features are extracted from the EEG/EMG signal in each epoch, which are fed to the six base classifiers. Three ensemble classifiers are created from DT and kNN using the random subspace and bagging methods. Each classifier classifies the epoch as either Wake, NREM, or REM, and outputs confidence scores for each class indicating how strongly it believes the epoch to be from that class. The confidence scores from the classifiers are averaged to form a consensus confidence score. The class with the greatest consensus confidence score is chosen as the final classification, unless the confidence score is below the rejection threshold, in which case it is left unscored.

Figure 2

Mean error rates of the base classifiers, as well as a second human scorer. The height of the entire bar indicates the error rate of the classifier. The portion in grey designates errors which the second human also made (“double-fault” errors), and the portion in black designates errors which the algorithm made but the second human did not (“unique” errors). The lower set of error bars display the SE of unique errors, while the higher set of error bars display the SE of all errors.

Figure 3

Ensemble classifiers. A) Base classifier DT compared with ensemble classifiers DT-Bag and DT-RS. B) Base classifier kNN compared with ensemble classifier kNN-RS. The portion in grey designates errors which the second human also made (“double-fault” errors), and the portion in black designates errors which the algorithm made but the second human did not (“unique” errors). The lower set of error bars display the SE of unique errors, while the higher set of error bars display the SE of all errors.

Figure 4

Error rate of MCS compared with SVM, the most accurate single classifier, and a second human scorer. SVM made significantly more errors than the second human scorer (p=0.007), but MCS did not. The portion in grey designates errors which the second human also made (“double-fault” errors), and the portion in black designates errors which the algorithm made but the second human did not (“unique” errors). The lower set of error bars display the SE of unique errors, while the higher set of error bars display the SE of all errors.

Figure 5

Rejections. A) Rejection-error curve displaying the trade-off between the proportion of epochs rejected and the error rate of the remaining epochs. B) Comparing MCS and SVM at two rejection rates, 0.02 and 0.10. The portion in grey designates errors which the second human also made (“double-fault” errors), and the portion in black designates errors which the algorithm made but the second human did not (“unique” errors). The lower set of error bars display the SE of unique errors, while the higher set of error bars display the SE of all errors.

Figure 6

Transitional epochs and error rate. (A) The error rate is represented by the entire bar, and the fraction of those errors which were transitional epochs is represented by the purple bars. Only sensitivity errors are considered. (B) The percentage of epochs which are transitional is represented by the entire bar, and the fraction of those epochs which were errors is represented by the purple bars. Error bars are 1 SE.