A Measurement-Based Protocol Improves Interrater Agreement and Accuracy of Right Ventricular Systolic Pressure Assessment by Echocardiography in Children: A Call for Quality Improvement

Abstract

Background: Echocardiographic measurements carry the promise of improving interrater (IR) agreement over subjective assessment. The aim of this study was to assess the effect of implementing a measurement-based protocol on IR agreement and accuracy in reporting of right ventricular (RV) systolic pressure in children. The effect of this reporting protocol on IR agreement in reporting RV dilation, hypertrophy, and systolic function was also evaluated.

Methods: Five echocardiography readers reported their assessment of RV systolic pressure, dilation, hypertrophy, and systolic function on 40 deidentified echocardiograms using their individual accustomed methods and then using an agreed-upon protocol based solely on RV measurements. IR agreement was assessed using κ statistics. Accuracy of RV systolic pressure ratings was assessed using the McNemar test in comparison with hemodynamic data obtained by cardiac catheterization. The reliability of the RV measurements was assessed using intraclass correlation coefficient (ICC) and coefficient of variation.

Results: IR agreement and accuracy of RV systolic pressure assessment improved after using the measurement-based protocol (agreement from 0.39 [95% CI, 0.27-0.5] to 0.62 [95% CI, 0.48-0.76] and accuracy from 18 of 40 to 29 of 40 [P = .03]). IR agreement of RV dilation improved (from 0.36 [95% CI, 0.25-0.48] to 0.63 [95% CI, 0.48-0.79]), while IR agreement of RV hypertrophy (from 0.29 [95% CI, 0.17-0.42] to 0.35 [95% CI, 0.15-0.55]) and RV systolic function (from 0.57 [95% CI, 0.45-0.69] to 0.53 [95% CI, 0.41-0.66]) did not improve. The reliability of the measurements was good (ICC > 0.8), except for RV free wall thickness (ICC = 0.62, coefficient of variation = 24%) and RV fractional area change (ICC = 0.47, coefficient of variation = 22%), suggesting a possible reason for the lack of improvement in IR agreement of RV hypertrophy and RV systolic function. Heteroscedasticity was observed in the reliability of RV measurements, with the ICC being significantly lower at larger magnitudes for all RV measurements.

Conclusions: Standardization of reporting protocols using RV measurements in place of subjective assessment improved IR agreement and accuracy of RV systolic pressure assessment. Reliable measurements (RV systolic pressure and dilation) resulted in improvement in IR agreement while unreliable measurements (RV hypertrophy and systolic function) did not. Special attention to measurements' reliability and heteroscedasticity of reliability is required when designing clinical protocols to decrease IR disagreement as a source of error.

Keywords: Interrater agreement; Pulmonary hypertension; Reliability; Right ventricle; Variability.