Inter- and Intra-Observer Variability and the Effect of Experience in Cine-MRI for Adhesion Detection

Abstract

Cine-MRI for adhesion detection is a promising novel modality that can help the large group of patients developing pain after abdominal surgery. Few studies into its diagnostic accuracy are available, and none address observer variability. This retrospective study explores the inter- and intra-observer variability, diagnostic accuracy, and the effect of experience. A total of 15 observers with a variety of experience reviewed 61 sagittal cine-MRI slices, placing box annotations with a confidence score at locations suspect for adhesions. Five observers reviewed the slices again one year later. Inter- and intra-observer variability are quantified using Fleiss' (inter) and Cohen's (intra) κ and percentage agreement. Diagnostic accuracy is quantified with receiver operating characteristic (ROC) analysis based on a consensus standard. Inter-observer Fleiss' κ values range from 0.04 to 0.34, showing poor to fair agreement. High general and cine-MRI experience led to significantly (p < 0.001) better agreement among observers. The intra-observer results show Cohen's κ values between 0.37 and 0.53 for all observers, except one with a low κ of -0.11. Group AUC scores lie between 0.66 and 0.72, with individual observers reaching 0.78. This study confirms that cine-MRI can diagnose adhesions, with respect to a radiologist consensus panel and shows that experience improves reading cine-MRI. Observers without specific experience adapt to this modality quickly after a short online tutorial. Observer agreement is fair at best and area under the receiver operating characteristic curve (AUC) scores leave room for improvement. Consistently interpreting this novel modality needs further research, for instance, by developing reporting guidelines or artificial intelligence-based methods.

Keywords: (MeSH); magnetic resonance imaging; observer variation; tissue adhesions.

Figure 1

Flowchart of the inclusion process for the observer dataset. A total of 20% of eligible patients was sampled randomly, after splitting them into groups with and without adhesions. All sagittal slices of these patients were included in the study, after exclusion due to insufficient quality or incomplete data from observers.

Figure 2

Three examples of cine-MRI slice with typical adhesions, with consensus annotations overlayed as blue boxes. (A) A slice without adhesions present, (B) a slice with an adhesion to the anterior abdominal wall, and (C) a slice with an adhesion in the pelvic area.

Figure 3

AUC versus: (A) Years of general experience, including residency; (B) amount of cine-MRI cases reviewed; and (C) amount of adhesion cine-MRI cases reviewed. The left column shows individual AUC scores per observer, with with orange squares indicating low experience grouping and blue circles indicating high experience grouping. The right column shows the diagonally averaged AUC for the experience groupings, with bars indicating 95% confidence intervals. This figure excludes the consensus observers since they set the reference standard.

Figure 4

Individual ROC curves for each observer. The title of each subplot shows the observer number, followed by their AUC score. This figure excludes the consensus observers since they set the reference standard.

Figure 5

Three examples of cine-MRI slice with varying levels of agreement between observers. Annotations are overlayed as boxes, with a unique color per observer. (A) A slice with high agreement for being negative (only two observers annotated), (B) a slice with high agreement for being positive in the same location, and (C) a slice with many positive annotations but in varying locations.