Interrater and intrarater agreement on heart murmurs

Abstract

Objective: To investigate interrater and intrarater agreement between physicians and medical students on heart sound classification from audio recordings, and factors predicting agreement with a reference classification.

Design: Intra- and interrater agreement study.

Subjects: Seventeen GPs and eight cardiologists from Norway and the Netherlands, eight medical students from Norway.

Main outcome measures: Proportion of agreement and kappa coefficients for intrarater agreement and agreement with a reference classification.

Results: The proportion of intrarater agreement on the presence of any murmur was 83% on average, with a median kappa of 0.64 (range k = 0.09-0.86) for all raters, and 0.65, 0.69, and 0.61 for GPs, cardiologist, and medical students, respectively.The proportion of agreement with the reference on any murmur was 81% on average, with a median kappa of 0.67 (range 0.29-0.90) for all raters, and 0.65, 0.69, and 0.51 for GPs, cardiologists, and medical students, respectively.Distinct murmur, more than five years of clinical practice, and cardiology specialty were most strongly associated with the agreement, with ORs of 2.41 (95% CI 1.63-3.58), 2.19 (1.58-3.04), and 2.53 (1.46-4.41), respectively.

Conclusion: We observed fair but variable agreement with a reference on heart murmurs, and physician experience and specialty, as well as murmur intensity, were the factors most strongly associated with agreement.Key points:Heart auscultation is the main physical examination of the heart, but we lack knowledge of inter- and intrarater agreement on heart sounds.• Physicians identified heart murmurs from heart sound recordings fairly reliably compared with a reference classification, and with fair intrarater agreement.• Both intrarater agreement and agreement with the reference showed considerable variation between doctors• Murmur intensity, more than five years in clinical practice, and cardiology specialty were most strongly linked to agreement with the reference.

Keywords: Interrater; agreement; heart auscultation; heart murmur; intrarater; valvular heart disease.

Figure 1.

Kappa values for agreement with the reference on the presence of murmur for each rater. Each point represents individual raters.

Figure 2.

Scatter plot showing correlation of the Kappa values between intrarater agreement on the x-axis and agreement with the reference classification on the y-axis.

Figure 3.

The proportion of agreement with the reference classification for each rater, on the level of ‘systolic/diastolic/normal’. The vertical line shows the overall mean proportion of agreement at 75%.