Defining the real-world reproducibility of visual grading of left ventricular function and visual estimation of left ventricular ejection fraction: impact of image quality, experience and accreditation

Abstract

Left ventricular function can be evaluated by qualitative grading and by eyeball estimation of ejection fraction (EF). We sought to define the reproducibility of these techniques, and how they are affected by image quality, experience and accreditation. Twenty apical four-chamber echocardiographic cine loops (Online Resource 1-20) of varying image quality and left ventricular function were anonymized and presented to 35 operators. Operators were asked to provide (1) a one-phrase grading of global systolic function (2) an "eyeball" EF estimate and (3) an image quality rating on a 0-100 visual analogue scale. Each observer viewed every loop twice unknowingly, a total of 1400 viewings. When grading LV function into five categories, an operator's chance of agreement with another operator was 50% and with themself on blinded re-presentation was 68%. Blinded eyeball LVEF re-estimates by the same operator had standard deviation (SD) of difference of 7.6 EF units, with the SD across operators averaging 8.3 EF units. Image quality, defined as the average of all operators' assessments, correlated with EF estimate variability (r = -0.616, p < 0.01) and visual grading agreement (r = 0.58, p < 0.01). However, operators' own single quality assessments were not a useful forewarning of their estimate being an outlier, partly because individual quality assessments had poor within-operator reproducibility (SD of difference 17.8). Reproducibility of visual grading of LV function and LVEF estimation is dependent on image quality, but individuals cannot themselves identify when poor image quality is disrupting their LV function estimate. Clinicians should not assume that patients changing in grade or in visually estimated EF have had a genuine clinical change.

Keywords: Echocardiography; Heart failure; Reproducibility of results; Ventricular function.

Fig. 1

Intra-operator self-disagreement by operators reassessing the same images blind to their previous assessment. The area of the bubbles represents the frequency of assessments with this combination

Fig. 2

Inter-operator disagreement by different operators reassessing the same images. Each of the 20 rows is a different case. The row shows a histogram of the 70 assessments made by the 35 operators for that case. In this figure, cases are ordered by the average grading given by operators

Fig. 3

Bland-Altman plots of first and second EF estimates when the same case was re-presented to the same operator. The left panel shows accredited operators. The right panel shows non-accredited operators. Paler blue points are an estimate where the operator judged the image of high quality, whereas dark blue represents a poor quality image. Larger dots indicate more experienced operators. In drawing this graph we have added a random ±1 % “jitter” so that multiple identical values may be appreciated

Fig. 4

Visual estimation of EF for 20 different cases, arranged from lowest average EF to highest average EF. Each column represents one echocardiographic case ordered from lowest average EF to highest average EF. The points in the column represent the EF estimates by 35 operators viewing the images twice. Paler blue points are an estimate where the operator judged the image of high quality, whereas dark blue represents a poor quality image. Larger dots indicate more experienced operators. In drawing this graph we have added a random ±1 % “jitter” so that multiple identical values may be appreciated

Fig. 5

Bland-Altman plot of intra-operator self-disagreement by operators reassessing the quality of the same images blind to their previous assessment. The left panel shows accredited operators. The right panel shows non-accredited operators. Larger dots indicate more experienced operators. In drawing this graph we have added a random ±1 “jitter” so that multiple identical values may be appreciated

Fig. 6

Inter-operator disagreement by operators assessing quality for the same images. Each column represents one echocardiographic case ordered from lowest average quality to highest average quality. The points in the column represent the assessment of quality by 35 operators viewing the images twice. Quality was assessed on a 0–100 scale. The cases are arranged from lowest to highest mean quality score. In drawing this graph we have added a random ±1 “jitter” so that multiple identical values may be appreciated

Fig. 7

Better image quality allows observers to agree with each other on ventricular function. Each point represents the proportion of visual grading assessments that agreed with commonest function assessment for that case versus the average quality score awarded by all observations of that case

Fig. 8

Impact of image quality on variability of EF estimates. Each point represents the standard deviation of estimates between operators for a case versus the mean quality estimate by all operators for that case

Fig. 9

Relationship between quality score and the number of categories’ deviation from the modal consensus of visual grading. Each point represents how many categories a single operator’s visual grading is from the modal visual grading versus the quality assessment the individual operator made at that time. The left panel shows accredited operators. The right panel shows non-accredited operators. Larger dots indicate more experienced operators

Fig. 10

Relationship between quality score given by an operator and the difference in their EF estimate from the mean of all operators. Each point represents the absolute difference a single operator’s EF estimate is from the mean of all operators versus the quality assessment the individual operator made at that time. The left panel shows accredited operators. The right panel shows non-accredited operators. Larger dots indicate more experienced operators