Assessing practical skills in cardiopulmonary resuscitation: Discrepancy between standard visual evaluation and a mechanical feedback device

Abstract

This paper aims to analyze agreement in the assessment of external chest compressions (ECC) by 3 human raters and dedicated feedback software.While 54 volunteer health workers (medical transport technicians), trained and experienced in cardiopulmonary resuscitation (CPR), performed a complete sequence of basic CPR maneuvers on a manikin incorporating feedback software (Laerdal PC v 4.2.1 Skill Reporting Software) (L), 3 expert CPR instructors (A, B, and C) visually assessed ECC, evaluating hand placement, compression depth, chest decompression, and rate. We analyzed the concordance among the raters (A, B, and C) and between the raters and L with Cohen's kappa coefficient (K), intraclass correlation coefficients (ICC), Bland-Altman plots, and survival-agreement plots.The agreement (expressed as Cohen's K and ICC) was ≥0.54 in only 3 instances and was ≤0.45 in more than half. Bland-Altman plots showed significant dispersion of the data. The survival-agreement plot showed a high degree of discordance between pairs of raters (A-L, B-L, and C-L) when the level of tolerance was set low.In visual assessment of ECC, there is a significant lack of agreement among accredited raters and significant dispersion and inconsistency in data, bringing into question the reliability and validity of this method of measurement.

Figure 1

Bland–Altman plots. Legend: A, B, and C represent the 3 human raters, and L represents the dedicated feedback software. Diff L-A, Diff L-B, and Diff L-C represent the differences between the score (0–100) for each human rater versus L. Mean L-A, mean L-B, and mean L-C represent the mean score (0–100) between each human rater and L.

Figure 2

Survival–agreement plot: Proportion of discordance between students’ scores awarded by raters and L. Legend: A, B, and C represent the 3 human raters, and L represents the dedicated feedback software. Diff L-A, Diff L-B, and Diff L-C represent the differences between the score (0–100) for each human rater versus L. The X-axis (absolute differences between human raters and L) represents the observed differences, and the Y-axis (discordance proportion) represents the proportion of cases with differences that are at least as large as the observed difference and therefore represents the proportion of discordant cases.