Inter-rater concordance and operating definitions of radiologic nodal feature assessment in human papillomavirus-positive oropharyngeal carcinoma

Abstract

Background and purpose: This study aims to evaluate the reliability of radiologic nodal feature assessment in clinical node-positive human papillomavirus-positive oropharyngeal carcinoma.

Materials and methods: Baseline CTs or MRIs of clinical node-positive human papillomavirus-positive oropharyngeal carcinoma diagnosed between 2012 and 2015 were reviewed independently by two neuroradiologists for seven nodal features: radiologic nodal involvement, cystic change, presence of necrosis, clustering, conglomeration, coalescence, and extranodal extension. Consensus operating definitions were derived after discussion. The features were re-reviewed in a randomly selected cohort. Levels of certainty (probability of presence: <25%, ∼50%, ∼75%, and >90%) were recorded. Interrater concordance was calculated using Cohen's kappa coefficient.

Results: A total of 413 patients (826 necks) were eligible. At initial review, the inter-rater kappa values for: radiologic nodal involvement, cystic change, necrosis, clustering, conglomeration, coalescence, and extranodal extension were 0.92, 0.64, 0.48, 0.32, 0.32, 0.62, and 0.56, respectively. A re-review of 94 randomly selected cases (188 necks) after consolidation of operating definitions for nodal features showed that the inter-rater kappa values of these features were 0.83, 0.62, 0.58, 0.32, 0.18, 0.68, and 0.74 when considering ≥50% certainty as positive, and improved to 0.94, 0.66, 0.59, 0.33, 0.19, 0.76, and 0.86 when considering ≥75% certainty as positive.

Conclusion: Clearly defined nomenclature results in improved interrater reliability when assessing radiologic nodal features, especially for coalescent adenopathy and extranodal extension. Higher levels of certainty are associated with higher inter-rater agreement. Radiology reporting should include clear definitions of clinically relevant nodal features as well as levels of certainty to serve various needs in clinical care and research.

Keywords: CT; Extranodal extension; Human papillomavirus; Inter-rater concordance; Lymph node; MRI; Oropharyngeal carcinoma; Radiologic nodal feature.