Meta-Analysis of Interobserver Agreement in Assessment of Interstitial Lung Disease Using High-Resolution CT

Abstract

Background High-resolution CT (HRCT) is central to the assessment of interstitial lung disease (ILD), and accurate classification of disease has important implications for patients. Evaluation of imaging features can be challenging, even for experienced thoracic radiologists. Previous work has provided equivocal evidence on the interpretation of HRCT features at ILD-related imaging. Purpose To perform a meta-analysis to assess the level of agreement among expert thoracic radiologists in interpreting ILD-related imaging. Materials and Methods A systematic literature search from January 2000 to October 2023 of the Ovid MEDLINE, Embase, and Cochrane Central Register of Controlled Trials databases was performed for articles reporting assessments of interobserver agreement between thoracic radiologists for evaluation of ILD findings, such as severity and progression of disease, presence of features such as honeycombing and ground-glass opacification, and classification based on the 2011 and 2018 American Thoracic Society/European Respiratory Society/Japanese Respiratory Society/Asociación Latinoamericana del Tórax (ATS/ERS/JRS/ALAT) guidelines for idiopathic pulmonary fibrosis (IPF). Meta-analysis was performed using a random-effects model to obtain pooled κ or intraclass correlation coefficient (ICC) values as measures of interobserver agreement. Results The final analysis included 13 studies consisting of 6943 images and 146 radiologists. In 10 studies assessing agreement of specific radiologic findings in ILD, the pooled κ value was 0.56 (95% CI: 0.43, 0.70). In eight studies, the assessed interobserver agreement of the ATS/ERS/JRS/ALAT diagnostic guidelines for IPF based on usual interstitial pneumonia (UIP) patterns, the pooled κ value was 0.61 (95% CI: 0.48, 0.74). One study reported a κ value of 0.87 for ILD progression. Seven studies assessing ILD severity could not be pooled; the individual κ values for ILD severity ranged from 0.64 to 0.90, and ICC values ranged from 0.63 to 0.96. Conclusion There was moderate agreement between thoracic radiologists when assessing ILD features and UIP pattern diagnosis but little evidence on agreement of disease severity, extent, or progression. Meta-analysis registry no. PROSPERO CRD42022361803 © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Humbert in this issue.

Graphical abstract

Figure 1:

Preferred Reporting Items for Systematic Reviews and Meta-Analyses, or PRISMA, flowchart. Ovid MEDLINE, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) searches yielded 4658 results. The full texts of 266 studies were reviewed, including backward reference searching. Of the 70 studies eligible for inclusion in the systematic review, 13 provided sufficient data for meta-analysis. AI = artificial intelligence, ILD = interstitial lung disease.

Figure 2:

Forest plot for 10 studies assessing interobserver agreement of specific disease patterns at high-resolution CT. The dotted line represents the pooled κ using the three-level random-effects model; boxes indicate the point estimate of the κ value for each study (study weight represented by the size of the box); horizontal bars show the 95% CI for each study; and the center of each diamond represents the overall pooled κ, while the width of the diamond indicates the 95% CI.

Figure 3:

Forest plot for studies assessing interobserver variability of the 2011 and 2018 American Thoracic Society/European Respiratory Society/Japanese Respiratory Society/Asociación Latinoamericana del Tórax (ATS/ERS/JRS/ALAT) diagnostic guidelines for radiologic features of usual interstitial pneumonia. Eight studies had sufficient data to be included in either the 2011 or 2018 subgroup analysis or both. The dotted line represents the pooled κ using the three-level random-effects model; boxes indicate the point estimate of the κ value for each study (study weight represented by the size of the box); horizontal bars show the 95% CI for each study; and the center of each diamond represents the overall pooled κ, while the width of the diamond indicates the 95% CI.