A large web-based observer reliability study of early ischaemic signs on computed tomography. The Acute Cerebral CT Evaluation of Stroke Study (ACCESS)

Abstract

Background: Early signs of ischaemic stroke on computerised tomography (CT) scanning are subtle but CT is the most widely available diagnostic test for stroke. Scoring methods that code for the extent of brain ischaemia may improve stroke diagnosis and quantification of the impact of ischaemia.

Methodology and principal findings: We showed CT scans from patients with acute ischaemic stroke (n = 32, with different patient characteristics and ischaemia signs) to doctors in stroke-related specialties world-wide over the web. CT scans were shown twice, randomly and blindly. Observers entered their scan readings, including early ischaemic signs by three scoring methods, into the web database. We compared observers' scorings to a reference standard neuroradiologist using area under receiver operator characteristic curve (AUC) analysis, Cronbach's alpha and logistic regression to determine the effect of scales, patient, scan and observer variables on detection of early ischaemic changes. Amongst 258 readers representing 33 nationalities and six specialties, the AUCs comparing readers with the reference standard detection of ischaemic signs were similar for all scales and both occasions. Being a neuroradiologist, slower scan reading, more pronounced ischaemic signs and later time to CT all improved detection of early ischaemic signs and agreement on the rating scales. Scan quality, stroke severity and number of years of training did not affect agreement.

Conclusions: Large-scale observer reliability studies are possible using web-based tools and inform routine practice. Slower scan reading and use of CT infarct rating scales improve detection of acute ischaemic signs and should be encouraged to improve stroke diagnosis.

Figure 1. Agreement between observers and the reference standard by observer specialty.

A box and whisker plot is shown for each of the 1/3 MCA, IST-3 and ASPECTS scales, with the observer groups listed on the left hand side. Each box and whisker represents the point estimate of the area under the curve (AUC, on x-axis, box) and 95% confidence intervals (whisker) for the observers in that group compared with the reference standard. A larger box indicates that there were more observers in that group. ER doctors  =  emergency doctors; GPs  =  general practitioners or family physicians.

Figure 2. Agreement between observers and the reference standard by scan features.

A box and whisker plot is shown for each of the 1/3 MCA, IST-3 and ASPECTS scales, with the scan features listed on the left hand side. Each box and whisker represents the point estimate of the area under the curve (AUC, on x-axis, box) and 95% confidence intervals (whisker) for the observers in that group compared with the reference standard.

Figure 3. “Bubble plots” show observers' score distributions for each scan for each scoring method.

The three scoring methods are the 1/3 MCA, IST-3 and ASPECTS scores. X axis indicates the individual scan identification numbers, the Y axis indicates the scores on each scale, and the scan quality as judged by the reference standard is indicated in blue (poor), moderate (yellow) or good (red).