Intra- and interreader reproducibility of PI-RADSv2: A multireader study

Abstract

Background: The Prostate Imaging Reporting and Data System version 2 (PI-RADSv2) has been in use since 2015; while interreader reproducibility has been studied, there has been a paucity of studies investigating the intrareader reproducibility of PI-RADSv2.

Purpose: To evaluate both intra- and interreader reproducibility of PI-RADSv2 in the assessment of intraprostatic lesions using multiparametric magnetic resonance imaging (mpMRI).

Study type: Retrospective.

Population/subjects: In all, 102 consecutive biopsy-naïve patients who underwent prostate MRI and subsequent MR/transrectal ultrasonography (MR/TRUS)-guided biopsy.

Field strength/sequences: Prostate mpMRI at 3T using endorectal with phased array surface coils (TW MRI, DW MRI with ADC maps and b2000 DW MRI, DCE MRI).

Assessment: Previously detected and biopsied lesions were scored by four readers from four different institutions using PI-RADSv2. Readers scored lesions during two readout rounds with a 4-week washout period.

Statistical tests: Kappa (κ) statistics and specific agreement (P_o ) were calculated to quantify intra- and interreader reproducibility of PI-RADSv2 scoring. Lesion measurement agreement was calculated using the intraclass correlation coefficient (ICC).

Results: Overall intrareader reproducibility was moderate to substantial (κ = 0.43-0.67, P_o = 0.60-0.77), while overall interreader reproducibility was poor to moderate (κ = 0.24, P_o = 46). Readers with more experience showed greater interreader reproducibility than readers with intermediate experience in the whole prostate (P = 0.026) and peripheral zone (P = 0.002). Sequence-specific interreader agreement for all readers was similar to the overall PI-RADSv2 score, with κ = 0.24, 0.24, and 0.23 and P_o = 0.47, 0.44, and 0.54 in T₂ -weighted, diffusion-weighted imaging (DWI), and dynamic contrast-enhanced (DCE), respectively. Overall intrareader and interreader ICC for lesion measurement was 0.82 and 0.71, respectively.

Data conclusion: PI-RADSv2 provides moderate intrareader reproducibility, poor interreader reproducibility, and moderate interreader lesion measurement reproducibility. These findings suggest a need for more standardized reader training in prostate MRI.

Level of evidence: 2 Technical Efficacy: Stage 2.

Keywords: MRI; PI-RADS; prostate cancer; reader reproducibility.

Figure 1.. Example of lesion identification presented to all readers.

Shown is a demarcated left apical peripheral zone lesion on T2W axial pulse sequence (A) with associated T2W axial and sagittal imaging, ADC map, calculated b2000 DWI, and DCE MR images. Readers were provided the full mpMRI image sets for evaluation and scoring was only completed on provided, pre-identified lesions.

Figure 2.. Overall PI-RADSv2 scoring distribution chart.

Each lesion (n=205) represented as a row with the frequency of scores reported for PI-RADSv2 categories. Lesions with consistent scoring, indicating high agreement, are represented by darker shades of red (max count = 4 readers x 2 rounds = 8). Lesions with poor agreement in scoring demonstrate lower frequencies across multiple PI-RADSv2 scoring categories. Two representative examples demonstrating high and low scoring agreement are shown with T2-weighted (T2), ADC, high b-value DWI (b2000) and DCE pulse sequences. Top – high agreement: A left base anterior peripheral zone lesion (Gleason Score 3+4 = 7) with reproducible scoring, in which all 4 readers assigned PI-RADSv2 Category 4 in both rounds. Bottom – low agreement: A right apical-mid peripheral zone lesion (Gleason Score 3+4 = 7) showing poor consensus, with two PI-RADSv2 Category 2 scores, four PI-RADSv2 Category 3 scores, and two PI-RADSv2 Category 4 scores. Two readers reported the same score in both rounds (Reader 1 PI-RADSv2 score 3, Reader 4 PI-RADSv2 score 2), with other readers demonstrating inconsistent scoring between round 1 and round 2. Prostate zonal location of each lesion and targeted biopsy results are indicated along the left side of the figure. PZ = peripheral zone, TZ = transition zone, and CZ = central zone.

Figure 3.

Intra-reader repeatability using Kappa (A) and specific agreement (B) among 4 readers and 2 rounds of lesions scoring. Readers 1 and 2 were moderate experience level, and readers 3 and 4 were expert experience level.

Figure 4.

Inter-reader reproducibility analysis of all, moderate and expert experience level readers using kappa (A) and specific agreement (B) statistics. *= significant value (P<.05) between moderate and expert level readers.

Figure 5.

Lesion coverage probability versus absolute change in lesion size measurement for all four readers in the whole prostate. For the average of four readers, 95% of all lesions had an absolute value ≤4.88 mm lesion measurement change. The shading that flanks each reader’s line represents 95% CIs.