Training and calibration improve inter-reader reliability of joint damage assessment using magnetic resonance image scoring and computerized erosion volume measurement

Abstract

Objective: To assess the inter-reader reliability of 3 rheumatologist readers before and after training using 2 methods of assessment: magnetic resonance imaging (MRI) computerized erosion volume assessment and MRI scoring using the OMERACT-5 Rheumatoid Arthritis MRI Score (OM-5 RAMRIS) criteria.

Methods: Erosion volumes were measured in 10 patients [5 wrist and 5 metacarpophalangeal (MCP) joint studies] with rheumatoid arthritis. Erosion scores were derived from this group and 8 additional subjects to provide a total of 18 subjects (10 wrist and 8 MCP joint studies) with MRI scores for comparison. Subjects were selected from existing MRI databases to provide a spectrum of joint damage for assessment. Initial reading was undertaken after the 2 inexperienced readers were provided with instructions regarding OSIRIS computer software and definitions of the OMERACT score; no other formal training was undertaken. One month after the initial reading, the 2 inexperienced readers undertook a 3 hour training session and all 3 readers then took part in 2 subsequent 2 hour calibration sessions. Each reader then reread the original MRI studies using the computerized erosion volume method and the OMERACT MRI RA score. The interval between the baseline and post-training reading was 2 months. All reading was undertaken on a computer workstation and readers were blinded to other readers' results.

Results: For the wrist MRI studies, inter-reader agreement improved considerably after training for both the computerized MRI volume method and the OMERACT MRI score [intraclass correlation coefficients (ICC) 0.21 and 0.46, respectively, pre-training; 0.92 and 0.85 post-training]. The correlation between all readers' scores and volumes was excellent at baseline and post-training. For the MCP joint studies, inter-reader agreement was moderate at baseline for the erosion volume and score (ICC 0.51 and 0.61). While there was some improvement in agreement post-training for the scoring method (ICC 0.75), there was no significant improvement for the erosion volumes (ICC 0.58).

Conclusion: Overall, inter-reader agreement for erosion scoring and volume measurement was higher for the wrist joint. The lack of improvement in the MCP joint region for the erosion volume measurements appears to relate primarily to difficulties in estimating the erosion border in the proximal MCP joints using the manual outlining tool. This limits the usefulness of erosion volume measurements in this joint region.