A Multicenter Assessment of Interreader Reliability of LI-RADS Version 2018 for MRI and CT

Abstract

Background Various limitations have impacted research evaluating reader agreement for Liver Imaging Reporting and Data System (LI-RADS). Purpose To assess reader agreement of LI-RADS in an international multicenter multireader setting using scrollable images. Materials and Methods This retrospective study used deidentified clinical multiphase CT and MRI and reports with at least one untreated observation from six institutions and three countries; only qualifying examinations were submitted. Examination dates were October 2017 to August 2018 at the coordinating center. One untreated observation per examination was randomly selected using observation identifiers, and its clinically assigned features were extracted from the report. The corresponding LI-RADS version 2018 category was computed as a rescored clinical read. Each examination was randomly assigned to two of 43 research readers who independently scored the observation. Agreement for an ordinal modified four-category LI-RADS scale (LR-1, definitely benign; LR-2, probably benign; LR-3, intermediate probability of malignancy; LR-4, probably hepatocellular carcinoma [HCC]; LR-5, definitely HCC; LR-M, probably malignant but not HCC specific; and LR-TIV, tumor in vein) was computed using intraclass correlation coefficients (ICCs). Agreement was also computed for dichotomized malignancy (LR-4, LR-5, LR-M, and LR-TIV), LR-5, and LR-M. Agreement was compared between research-versus-research reads and research-versus-clinical reads. Results The study population consisted of 484 patients (mean age, 62 years ± 10 [SD]; 156 women; 93 CT examinations, 391 MRI examinations). ICCs for ordinal LI-RADS, dichotomized malignancy, LR-5, and LR-M were 0.68 (95% CI: 0.61, 0.73), 0.63 (95% CI: 0.55, 0.70), 0.58 (95% CI: 0.50, 0.66), and 0.46 (95% CI: 0.31, 0.61) respectively. Research-versus-research reader agreement was higher than research-versus-clinical agreement for modified four-category LI-RADS (ICC, 0.68 vs 0.62, respectively; P = .03) and for dichotomized malignancy (ICC, 0.63 vs 0.53, respectively; P = .005), but not for LR-5 (P = .14) or LR-M (P = .94). Conclusion There was moderate agreement for LI-RADS version 2018 overall. For some comparisons, research-versus-research reader agreement was higher than research-versus-clinical reader agreement, indicating differences between the clinical and research environments that warrant further study. © RSNA, 2023 Supplemental material is available for this article. See also the editorials by Johnson and Galgano and Smith in this issue.

Graphical abstract

Figure 1:

Schematic of the retrospective study design. Deidentified examinations from the coordinating site and five other submitting sites were randomly assigned to two of 43 research readers for research reads. Features and categories were extracted from the clinical reports. This permitted the computation of interreader agreement between the research readers (R vs R) and between the research and clinical readers (R vs C). Twenty percent of images were also read twice by one of the research readers to permit the computation of intrareader agreement (R’ vs R’). DICOM = Digital Imaging and Communications in Medicine, LI-RADS = Liver Imaging Reporting and Data System, PHI = protected health information..

Figure 2:

Plot shows intraclass correlation coefficient (ICC) reader agreement for modified four-category Liver Imaging Reporting and Data System (LI-RADS) version 2018 scale based on imaging modality. Agreement among research reads only (research-research; ■) and between research and clinical reads (research-clinical; □) are shown. Tails represent 95% CIs. * P value < .05 by nonparametric bootstrap with per-case resampling. Research-versus-research agreement pooled over both modalities and for MRI only was better than research-versus-clinical agreement.

Figure 3:

MRI scans show (A) reader disagreement and (B) reader agreement. (A) Gadoxetic acid–enhanced MRI scans in a 56-year-old male patient with cirrhosis secondary to hepatitis C. From left to right: contrast-unenhanced (Pre), arterial phase (AP), portal venous phase (PVP), and hepatobiliary phase (HBP) images. This 21-mm hepatobiliary phase hypointense observation (arrow) was characterized on the clinical read as having nonrim arterial phase hyperenhancement and washout appearance and was categorized as Liver Imaging Reporting and Data System (LI-RADS) category LR-5 (definitely hepatocellular carcinoma [HCC]). The first research reader characterized it as having a targetoid appearance and categorized it as LR-M (probably or definitely malignant, not specific for HCC). The second research reader characterized it as having no major features and paralleling the blood pool and categorized it as LR-2 (probably benign). It was subsequently resected and found to be a well-differentiated HCC. (B) Extracellular contrast–enhanced MRI scans in a 61-year-old female patient with cirrhosis secondary to hepatitis C. From left to right: contrast-unenhanced, arterial phase, portal venous phase, and delayed-phase (DP) images. This 31-mm observation (arrow) in the caudate lobe was characterized on the clinical read as having arterial phase hyperenhancement, washout appearance, and capsule appearance, and was categorized as LI-RADS category LR-5 (definitely HCC). Both research readers also categorized this observation as LR-5. The patient died of intracranial hemorrhage a few months later.

Figure 4:

Plot shows intraclass correlation coefficient (ICC) reader agreement for dichotomized classification of Liver Imaging Reporting and Data System (LI-RADS) version 2018 for the following dichotomized categories: probably or definitely malignant versus other, LR-5 (definitely hepatocellular carcinoma [HCC]) versus other, and LR-M (probably or definitely malignant, not specific for HCC) versus other. Agreement among research reads only (research-research; ■) and between research and clinical reads (research-clinical; □) are shown. Tails represent 95% CIs. * P < .05 by nonparametric bootstrap with per-case resampling. Research-research agreement for malignant categories was better than research-clinical agreement.

Figure 5:

Plot shows intraclass correlation coefficient (ICC) reader agreement for Liver Imaging Reporting and Data System (LI-RADS) version 2018 arterial phase hyperenhancement (APHE), washout, and capsule for research reads only (research-research; ■) and between research and clinical reads (research-clinical; □). Tails represent 95% CIs. No differences in ICCs between research-versus-research reads compared with researcher-versus-clinical reads were observed.

Figure 6:

Plot shows intraclass correlation coefficient (ICC) reader agreement for Liver Imaging Reporting and Data System (LI-RADS) version 2018 ancillary features with sufficient frequency for analysis, which included restricted diffusion, mild-moderate T2 hyperintensity, transitional phase hypointensity, and hepatobiliary phase hypointensity. These four features are only visible at MRI. Agreement among research reads only (research-research; ■) and between research and clinical reads (research-clinical; □) are shown. Tails represent 95% CIs. * P < .05 by nonparametric bootstrap with per-case resampling. Research-versus-research agreement for mild-moderate T2 hyperintensity was better than research-versus-clinical agreement.