Wide Inter-institutional Variation in Performance of a Molecular Classifier for Indeterminate Thyroid Nodules

Abstract

Background: The Afirma gene expression classifier (GEC) is used to assess malignancy risk in indeterminate thyroid nodules (ITNs) classified as Bethesda category III/IV. Our objective was to analyze GEC performance at two institutions with high thyroid cytopathology volumes but differing prevalence of malignancy.

Methods: Retrospective analysis of all ITNs evaluated with the GEC at Memorial Sloan Kettering Cancer Center (MSK; n = 94) and Mount Sinai Beth Israel (MSBI; n = 71). These institutions have differing prevalences of malignancy in ITNs: 30-38 % (MSK) and 10-19 % (MSBI). Surgical pathology was correlated with GEC findings for each matched nodule. Performance characteristics were estimated using Bayes Theorem.

Results: Patient and nodule characteristics were similar at MSK and MSBI. The GEC-benign call rates were 38.3 % (MSK) and 52.1 % (MSBI). Of the GEC-benign nodules, 8.3 % (MSK) and 13.5 % (MSBI) were treated surgically. Surgical pathology indicated that all of GEC-benign nodules were benign. Of the GEC-suspicious nodules, 60.0 % (MSK) and 61.7 % (MSBI) underwent surgery. Positive predictive values (PPVs) for GEC-suspicious results were 57.1 % (95 % CI 41.0-72.3) at MSK and 14.3 % (95 % CI 0.2-30.2) at MSBI. The estimated negative predictive values (NPVs) were 86-92 % at MSK and 95-98 % at MSBI.

Conclusions: There were wide variations in the Afirma GEC-benign call rate, PPV, and NPV between MSBI (a comprehensive health system) and MSK (a tertiary referral cancer center), which had differing rates of malignancy in ITNs. The GEC could not routinely alter management in either institution. We believe that this assay would be expected to be most informative in practice settings where the prevalence of malignancy is 15-21 %, such that NPV >95 % and PPV >25 % would be anticipated. Knowing the prevalence of malignancy in ITNs at a particular institution is critical for reliable interpretation of GEC results.

FIG. 1

a Estimated gene expression classifier (GEC) performance characteristics based on the prevalence of malignancy, with GEC sensitivity of 90 % and specificity 51 %. PPV positive predictive value; NPV negative predictive value, BCR benign call rate (percentage of indeterminate nodules that are GEC-benign). b Actual benign call rate (BCR) and PPV values at Memorial Sloan Kettering Cancer Center (MSK) and Mount Sinai Beth Israel (MSBI)

FIG. 2

Management of indeterminate thyroid nodules at (a) MSK and (b) MSBI. ITN indeterminate thyroid nodules, GEC gene expression classifier, internal patients undergoing workup and Afirma evaluation at MSK or MSBI, in contrast to patients referred already with workup and Afirma evaluation completed, FVPTC follicular variant of papillary thyroid carcinoma, PTC papillary thyroid carcinoma, Poorly diff. poorly differentiated, HCC Hürthle cell carcinoma, MTC medullary thyroid carcinoma

FIG. 3

Comparative pre-test and post-test probability of malignancy. Pre-test probabilities of malignancy at MSK and MSBI were substantially different from those of Alexander et al., leading to wide variation in GEC clinical performance. Post-test probability (negative test) represents the probability of malignancy given a GEC-benign result, or 1–NPV. Post-test probability (positive test) represents the probability of malignancy given a GEC-suspicious result, or PPV