Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules

Abstract

Objective: To evaluate the interobserver variability and performance in the interpretation of ultrasonographic (US) findings of thyroid nodules.

Materials and methods: 72 malignant nodules and 61 benign nodules were enrolled as part of this study. Five faculty radiologists and four residents independently performed a retrospective analysis of the US images. The observers received one training session after the first interpretation and then performed a secondary interpretation. Agreement was analyzed by Cohen's kappa statistic. Degree of performance was analyzed using receiver operating characteristic (ROC) curves.

Results: Agreement between the faculties was fair-to-good for all criteria; however, between residents, agreement was poor-to-fair. The area under the ROC curves was 0.72, 0.62, and 0.60 for the faculties, senior residents, and junior residents, respectively. There was a significant difference in performance between the faculties and the residents (p < 0.05). There was a significant increase in the agreement for some criteria in the faculties and the senior residents after the training session, but no significant increase in the junior residents.

Conclusion: Independent reporting of thyroid US performed by residents is undesirable. A continuous and specialized resident training is essential to enhance the degree of agreement and performance.

Keywords: Faculty; Internship and Residency; Observer variation; Thyroid Nodule; Ultrasonography.

Fig. 1

Right papillary carcinoma in 47-year-old woman. Axial (A) and sagittal (B) images representing case that showed moderate interobserver agreement for presence of microcalcifications. This case was described as having microcalcifications by all faculty radiologists and three out of four residents. One resident described it as having macrocalcifications.

Fig. 2

Left papillary carcinoma in 42-year-old man. Axial (A) and sagittal (B) images representing case that showed poor interobserver agreement for echotexture. At primary interpretation, three of five faculties and two of four residents described this case as having heterogeneous echotexture. Remaining observers described carcinoma as being homogeneous. At secondary interpretation, all faculties agreed on heterogeneous echotexture. In residents, however, no changes were seen.

Fig. 3

Left follicular adenoma in 18-year-old woman. Axial (A) and sagittal (B) images representing case that showed fair interobserver agreement for category. Faculty radiologists classified this case as indeterminate nodule at both primary and secondary interpretation. At primary interpretation, senior residents classified case as indeterminate (n = 1) and probably benign (n = 1). At secondary interpretation, however, they all classified case as indeterminate. Junior residents classified case as indeterminate and suspicious malignant at both primary and secondary interpretation.

Fig. 4

Receiver operating characteristic curves for faculty, senior residents, and junior residents. Area under curves was 0.72 for faculty, 0.62 for senior residents and 0.60 for junior residents. There is significant difference between area under the receiver operating characteristic curves values of faculties and residents.