PSG Validation of minute-to-minute scoring for sleep and wake periods in a consumer wearable device

Abstract

Background: Actigraphs are wrist-worn devices that record tri-axial accelerometry data used clinically and in research studies. The expense of research-grade actigraphs, however, limit their widespread adoption, especially in clinical settings. Tri-axial accelerometer-based consumer wearable devices have gained worldwide popularity and hold potential for a cost-effective alternative. The lack of independent validation of minute-to-minute accelerometer data with polysomnographic data or even research-grade actigraphs, as well as access to raw data has hindered the utility and acceptance of consumer-grade actigraphs.

Methods: Sleep clinic patients wore a consumer-grade wearable (Huami Arc) on their non-dominant wrist while undergoing an overnight polysomnography (PSG) study. The sample was split into two, 20 in a training group and 21 in a testing group. In addition to the Arc, the testing group also wore a research-grade actigraph (Philips Actiwatch Spectrum). Sleep was scored for each 60-s epoch on both devices using the Cole-Kripke algorithm.

Results: Based on analysis of our training group, Arc and PSG data were aligned best when a threshold of 10 units was used to examine the Arc data. Using this threshold value in our testing group, the Arc has an accuracy of 90.3%±4.3%, sleep sensitivity (or wake specificity) of 95.5%±3.5%, and sleep specificity (wake sensitivity) of 55.6%±22.7%. Compared to PSG, Actiwatch has an accuracy of 88.7%±4.5%, sleep sensitivity of 92.6%±5.2%, and sleep specificity of 60.5%±20.2%, comparable to that observed in the Arc.

Conclusions: An optimized sleep/wake threshold value was identified for a consumer-grade wearable Arc trained by PSG data. By applying this sleep/wake threshold value for Arc generated accelerometer data, when compared to PSG, sleep and wake estimates were adequate and comparable to those generated by a clinical-grade actigraph. As with other actigraphs, sleep specificity plateaus due to limitations in distinguishing wake without movement from sleep. Further studies are needed to evaluate the Arc's ability to differentiate between sleep and wake using other sources of data available from the Arc, such as high resolution accelerometry and photoplethysmography.

Fig 1. A receiver operating characteristic (ROC) curve showing sensitivity and specificity of Arc-derived sleep/wake scoring with PSG-derived sleep/wake scoring in the training group.

Each point represents a different threshold.

Fig 2

a-b. Bland Altman plots showing the difference between the Arc and PSG plotted against the mean for both total sleep time (TST) (2a) and wake after sleep onset (WASO) (2b) for each individual. Biases are marked and the dotted lines represent the upper and lower agreement limits of the biases.

Fig 3

a-c: Box plots comparing the accuracy (3a), sensitivity (3b), and specificity (3c) of each actigraph device compared to the polysomnogram. Data from the Actiwatch device in the Test group was analyzed separately using all four threshold options offered by the software.

Fig 4

a-b: Box plots comparing the difference of each actigraph device compared to the polysomnogram for total sleep time (4a) and wake after sleep onset (4b). Data from the Actiwatch device was analyzed separately using all four threshold options offered by the manufacturer’s software.