Observer agreement for small bowel ultrasound in Crohn's disease: results from the METRIC trial

Abstract

Purpose: To prospectively evaluate interobserver agreement for small bowel ultrasound (SBUS) in newly diagnosed and relapsing Crohn's disease.

Methods: A subset of patients recruited to a prospective trial comparing the diagnostic accuracy of MR enterography and SBUS underwent a second SBUS performed by one of a pool of six practitioners, who recorded the presence, activity and location of small bowel and colonic disease. Detailed segmental mural and extra-mural observations were also scored. Interobserver variability was expressed as percentage agreement with a construct reference standard, split by patient cohort, grouping disease as present or absent. Prevalence adjusted bias adjusted kappa (PABAK), and simple percentage agreement between practitioners, irrespective of the reference standard, were calculated.

Results: Thirty-eight patients (11 new diagnosis, 27 relapse) were recruited from two sites. Overall percentage agreement for small bowel disease presence against the consensus reference was 82% (52-95% (95%CI)), kappa coefficient (κ) 0.64, (substantial agreement) for new diagnosis and 81%, κ 0.63 (substantial agreement) for the relapsing cohort. Agreement for colonic disease presence was 64%, κ 0.27 (fair agreement) in new diagnosis and 78%,κ 0.56 (moderate agreement) in the relapsing cohort. Simple agreement between practitioners was 84% and 87% for small bowel and colonic disease presence respectively. Practitioners agreed on small bowel disease activity in 24/27 (89%) where both identified disease. Kappa agreement for detailed mural observations ranged from κ 0.00 to 1.00.

Conclusion: There is substantial practitioner agreement for small bowel disease presence in newly diagnosed and relapsing CD patients, supporting wider dissemination of enteric US.

Keywords: Crohn disease; Observer variation; Prospective studies; Ultrasonography.

Fig. 1

Presence of small bowel Crohn’s disease: Agreement between two reads and consensus reference. Number of patients are shown by segment and disease status (DP disease positive, DN disease negative), where two reads (dark blue), one read (light blue) and none of the reads agree (pale blue) with the consensus

Fig. 2

Agreement of first and second reads for disease location compared to consensus reference. The 1st read is shown in red with a circle symbol and the number of patients at the disease location. The 2nd read is shown in blue, using a cross symbol. The diagonal line indicates where reads agree with the consensus. For example, three patients were found to have disease in the Ileum (Il) by the consensus reference. The 1st read agreed with the consensus for two patients (red circle on diagonal numbered 2) and identified one patient with Terminal Ileum (TI) & Ileum (Il) (red circle numbered 1). The 2nd read in blue agreed with the consensus for all three patients (blue cross on diagonal numbered 3). Disease presence: D&TI Duodenum & Terminal Ileum, J Jejunum, Il Ileum, TI Terminal Ileum, TI&Il Terminal Ileum and Ileum, ND Disease Negative