Observer variation in the interpretation of intra-arterial angiograms and the risk of inappropriate decisions about carotid endarterectomy

Abstract

Objective: To determine how often observer variation in the interpretation of intra-arterial angiograms might alter the decision whether or not to refer patients for carotid surgery.

Methods: A prospective study was carried out in a consecutive series of 99 patients with transient ischaemic attacks and minor strokes. Interpretable angiographic films were available for 179 carotid artery bifurcations. Stenosis of the internal carotid artery was measured using mm scales, independently by three different radiologists (A, B, and C), using the European Carotid Surgery Trial method.

Results: An analysis of the grouped data showed good to moderate agreement by kappa statistics for radiologists A v B, B v C, and A v C of 0.68, 0.60, and 0.70 respectively. The mean absolute difference in the estimate of stenosis by each of the different radiologists (interobserver variation) was 9.5% and for each radiologist on two separate occasions (intraobserver variation) 8.4%. The degree of observer error was smallest among severely stenosed arteries. Although the absolute differences were small, "clinically important" differences which could change the treatment recommended from surgery to no surgery (or vice versa) occurred between radiologists A and B, B and C, and A and C in: seven (3.9%), six (3.4%), and 11 (6.1%) vessels respectively.

Conclusions: Because observer variation affects all of the imaging methods (Doppler, duplex, contrast arteriography, and MR angiography) used to select patients with transient ischaemic attack and stroke, these findings are likely to be widely relevant. Any centre assessing patients with cerebrovascular disease will need to implement strict quality control measures in the interpretation of angiograms (and other vascular imaging procedures) to minimise observer error and thereby reduce the number of inappropriate decisions made to refer for carotid artery surgery or not.