Investigating the accuracy of blood oxygen saturation measurements in common consumer smartwatches

Abstract

Blood oxygen saturation (SpO2) is an important measurement for monitoring patients with acute and chronic conditions that are associated with low blood oxygen levels. While smartwatches may provide a new method for continuous and unobtrusive SpO2 monitoring, it is necessary to understand their accuracy and limitations to ensure that they are used in a fit-for-purpose manner. To determine whether the accuracy of and ability to take SpO2 measurements from consumer smartwatches is different by device type and/or by skin tone, our study recruited patients aged 18-85 years old, with and without chronic pulmonary disease, who were able to provide informed consent. The mean absolute error (MAE), mean directional error (MDE) and root mean squared error (RMSE) were used to evaluate the accuracy of the smartwatches as compared to a clinical grade pulse oximeter. The percent of data unobtainable due to inability of the smartwatch to record SpO2 (missingness) was used to evaluate the measurability of SpO2 from the smartwatches. Skin tones were quantified based on the Fitzpatrick (FP) scale and Individual Typology Angle (ITA), a continuous measure of skin tone. A total of 49 individuals (18 female) were enrolled and completed the study. Using a clinical-grade pulse oximeter as the reference standard, there were statistically significant differences in accuracy between devices, with Apple Watch Series 7 having measurements closest to the reference standard (MAE = 2.2%, MDE = -0.4%, RMSE = 2.9%) and the Garmin Venu 2s having measurements farthest from the reference standard (MAE = 5.8%, MDE = 5.5%, RMSE = 6.7%). There were also significant differences in measurability across devices, with the highest data presence from the Apple Watch Series 7 (88.9% of attempted measurements were successful) and the highest data missingness from the Withings ScanWatch (only 69.5% of attempted measurements were successful). The MAE, RMSE and missingness did not vary significantly across FP skin tone groups, however, there may be a relationship between FP skin tone and MDE (intercept = 0.04, beta coefficient = 0.47, p = 0.04). No statistically significant difference was found between skin tone as measured by ITA and MAE, MDE, RMSE or missingness.

Fig 1

Accuracy of SpO₂ measurements against the FDA-cleared Masimo MightySat reference standard for A) Apple Watch Series 7, B) Garmin Venu 2S, C) Garmin Fenix 6 Pro, and D) Withings ScanWatch. Data falls into the categories of overestimate (red), within error range, meaning the difference of readings between reference reading and device reading is within a 2% error range, (blue), or underestimate (green). The dotted lines on the graph represent the 2% error of the clinical-grade MightySat Rx. Additionally, the relative percentage of data falling into the categories of overestimate (red), within error range (blue), underestimate (green), and missing (gray) are summarized in Fig 1.

Fig 2. Relative percentages of data falling into the categories of overestimate (red), Within error range (blue), underestimate (green), and missing (grey) for the Apple Watch Series 7, Garmin Venu 2S, Garmin Fenix 6 Pro, and Withings ScanWatch.

The black numbers above each bar represent the number of measurements within the category.

Fig 3

(A) Mean Directional Error (MDE) (B) Mean Absolute Error (MAE) (C) Root Mean Square Error (RMSE) (D) Missingness across skin tones classified by Fitzpatrick scale.

Fig 4

(A) Mean Absolute Error (MAE), (B) Mean Directional Error (MDE), (C) Root Mean Square Error (RMSE) and (D) Missingness across four commercial wearable devices by FP skin tone group.

Fig 5

Graphical abstract of the study, including the four devices that were tested (A), the reference standard (B), the role of device type on the accuracy and missingness of SpO2 values (C), and the role of skin tone on the accuracy of SpO2 values (D).