Performance of an automated polysomnography scoring system versus computer-assisted manual scoring

Abstract

Study objectives: Manual scoring of polysomnograms (PSG) is labor intensive and has considerable variance between scorers. Automation of scoring could reduce cost and improve reproducibility. The purpose of this study was to compare a new automated scoring system (YST-Limited, Winnipeg, Canada) with computer-assisted manual scoring.

Design: Technical assessment.

Setting: Five academic medical centers.

Participants: N/A.

Interventions: N/A.

Measurements and results: Seventy PSG files were selected at University of Pennsylvania (Penn) and distributed to five US academic sleep centers. Two blinded technologists from each center scored each file. Automatic scoring was performed at Penn by a YST Limited technician using a laptop containing the software. Variables examined were sleep stages, arousals, and apnea-hypopnea index (AHI) using three methods of identifying hypopneas. Automatic scores were not edited and were compared to the average scores of the 10 technologists. Intraclass correlation coefficient (ICC) was obtained for the 70 pairs and compared to across-sites ICCs for manually scored results. ICCs for automatic versus manual scoring were > 0.8 for total sleep time, stage N2, and nonrapid eye movement arousals and > 0.9 for AHI scored by primary and secondary American Academy of Sleep Medicine criteria. ICCs for other variables were not as high but were comparable to the across-site ICCs for manually scored results.

Conclusion: The automatic system yielded results that were similar to those obtained by experienced technologists. Very good ICCs were obtained for many primary PSG outcome measures. This automated scoring software, particularly if supplemented with manual editing, may increase laboratory efficiency and standardize PSG scoring results within and across sleep centers.

Keywords: apnea-hypopnea index; lung; polysomnography; reliability; scoring; sleep.

Figure 1

Bland-Altman plots showing the difference between autoscoring and the average of 10 scorers versus the average of the two scores, for total apnea-hypopnea index (AHI) scored by the American Academy of Sleep Medicine (AASM) primary criteria (top left), REM AHI scored by the AASM primary criteria (top right), number of central apneas (bottom left), and number of obstructive apneas (bottom right). ICC, intraclass correlation coefficient; REM, rapid eye movement sleep. The solid horizontal line in each panel represents the median difference and the two dashed lines represent the fifth and 95^th percentiles of the difference.

Figure 2

Bland-Altman plots showing the difference between autoscoring and the average of 10 scorers versus the average of the two scores, for total apnea-hypopnea index (AHI) scored by the American Academy of Sleep Medicine (AASM) secondary criteria (top left), REM AHI scored by the AASM secondary criteria (top right), total AHI scored by the research criteria (bottom left), and REM AHI scored by the research criteria (bottom right). ICC, intraclass correlation coefficient; REM, rapid eye movement sleep. The solid horizontal line in each panel represents the median difference and the two dashed lines represent the fifth and 95^th percentiles of the difference.

Figure 3

Bland-Altman plots showing the difference between autoscoring and the average of 10 scorers versus the average of the two scores, for sleep stages N1 (top left), N2 (top right), N3 (bottom left), and rapid eye movement (REM) sleep (bottom right). ICC, intraclass correlation coefficient. The solid horizontal line in each panel represents the median difference and the two dashed lines represent the fifth and 95^th percentiles of the difference.

Figure 4

Bland-Altman plots showing the difference between autoscoring and the average of 10 scorers versus the average of the two scores, for total sleep time (top left), onset of rapid eye movement (REM) sleep (top right), number of arousals in non-REM (bottom left), and REM sleep (bottom right). ICC, intraclass correlation coefficient. The solid horizontal line in each panel represents the median difference and the two dashed lines represent the fifth and 95^th percentiles of the difference.

Figure 5

Scatterplots of intraclass correlation coefficients (ICC) for autoscoring versus average manual scoring of the 17 variables measured (ordinate) versus within-site (A) and across-sites (B) ICCs for the same variables. Each dot is a separate variable. Diagonal lines are the lines of identity.