Agreement in computer-assisted manual scoring of polysomnograms across sleep centers

Abstract

Study objectives: To determine intersite agreement in respiratory event scoring of polysomnograms (PSGs) using different hypopnea definitions.

Design: Technical assessment.

Setting: Five academic medical centers.

Participants: N/A.

Interventions: N/A.

Measurements and results: Seventy good-quality PSGs performed in middle-aged women were manually scored by two experienced technologists at each of the five sleep centers using the particular laboratory's own software system. Studies were scored once by each scorer using American Academy of Sleep Medicine (AASM) standards for scoring sleep stages, arousals, and apneas. Hypopneas were then scored using three different AASM criteria: recommended, alternate, and research (Chicago). Means of each PSG variable for the scorers at each site were used to calculate an across-site intraclass correlation coefficient (ICC). Average AHI across the 10 scorers was 7.4 ± 12.3 (standard deviation) events/h using recommended criteria (ICC 0.984; 95% confidence interval [CI] 0.977-0.990), 12.1 ± 13.3 events/h using alternate criteria (ICC 0.947; 95% CI 0.889-0.972), and 15.1 ± 13.9 events/h with Chicago criteria (ICC 0.800; 95% CI 0.768-0.828). ICC across sites was 0.870 (95% CI = 0.847-0.889) for total sleep time, 0.861 (95% CI 0.837-0.881) for number of obstructive apneas and 0.683 (95% CI 0.640-0.722) for number of central apneas. ICCs across sites for hypopneas were very good using recommended criteria (ICC 0.843; 95% CI 0.820-0.870) but decreased when alternate criteria (ICC 0.728; 95% CI 0.689-0.763) and Chicago criteria (ICC 0.535; 95% CI 0.485-0.583) were used.

Conclusion: Experienced scorers at different laboratories have very good agreement in hypopnea and AHI results when good-quality PSGs are scored using AASM-recommended criteria. Substantial degradation of reliability was observed for alternative definitions of hypopneas, particularly that proposed for research.

Keywords: Apnea-hypopnea index; polysomnography; reliability; scoring.

Figure 1

Percentage of patients without obstructive sleep apnea (no OSA; apnea-hypopnea index [AHI] < five events/h), mild OSA (five ≤ AHI < 15), moderate 15 ≤ AHI < 30), and severe OSA (AHI ≥ 30) using the American Academy of Sleep Medicine (AASM)-recommended, AASM alternate, and Chicago criteria. Results based on the average AHI using each of the three scoring criteria across all 10 scorers.