Predicting Lymph Node Metastasis in Non-small Cell Lung Cancer: Prospective External and Temporal Validation of the HAL and HOMER Models

Abstract

Background: Two models, the Help with the Assessment of Adenopathy in Lung cancer (HAL) and Help with Oncologic Mediastinal Evaluation for Radiation (HOMER), were recently developed to estimate the probability of nodal disease in patients with non-small cell lung cancer (NSCLC) as determined by endobronchial ultrasound-transbronchial needle aspiration (EBUS-TBNA). The objective of this study was to prospectively externally validate both models at multiple centers.

Research question: Are the HAL and HOMER models valid across multiple centers?

Study design and methods: This multicenter prospective observational cohort study enrolled consecutive patients with PET-CT clinical-radiographic stages T1-3, N0-3, M0 NSCLC undergoing EBUS-TBNA staging. HOMER was used to predict the probability of N0 vs N1 vs N2 or N3 (N2|3) disease, and HAL was used to predict the probability of N2|3 (vs N0 or N1) disease. Model discrimination was assessed using the area under the receiver operating characteristics curve (ROC-AUC), and calibration was assessed using the Brier score, calibration plots, and the Hosmer-Lemeshow test.

Results: Thirteen centers enrolled 1,799 patients. HAL and HOMER demonstrated good discrimination: HAL ROC-AUC = 0.873 (95%CI, 0.856-0.891) and HOMER ROC-AUC = 0.837 (95%CI, 0.814-0.859) for predicting N1 disease or higher (N1|2|3) and 0.876 (95%CI, 0.855-0.897) for predicting N2|3 disease. Brier scores were 0.117 and 0.349, respectively. Calibration plots demonstrated good calibration for both models. For HAL, the difference between forecast and observed probability of N2|3 disease was +0.012; for HOMER, the difference for N1|2|3 was -0.018 and for N2|3 was +0.002. The Hosmer-Lemeshow test was significant for both models (P = .034 and .002), indicating a small but statistically significant calibration error.

Interpretation: HAL and HOMER demonstrated good discrimination and calibration in multiple centers. Although calibration error was present, the magnitude of the error is small, such that the models are informative.

Keywords: endobronchial ultrasound; lung cancer; lung cancer staging; mediastinal adenopathy.

Figure 1

Receiver operating characteristic curves of the HAL model in the combined validation cohort. The figure plots the area under the curve (AUC) for A, N2 or N3 (vs N0 or N1) disease (AUC = 0.873) for all patients that met inclusion criteria from all 13 institutions (n = 1,799), and for B, N2 or N3 (vs N0 or N1) disease (AUC = 0.875) for patients that met inclusion criteria from institutions that had more than 50 eligible patients (n = 1,701). HAL = Help with the Assessment of Adenopathy in Lung cancer model.

Figure 2

Observed vs predicted plots for the HAL model in the combined validation cohort for A, all patients that met inclusion criteria from all 13 institutions (n = 1,799) and B, for patients that met inclusion criteria from institutions that had more than 50 eligible patients (n = 1,701). The figure plots the probability of N2 or N3 (vs N0 or N1) disease by decile of expected risk as a function of the actual observed risk in that group. The observed probability for each decile is on the vertical axis; the predicted probability for each decile is on the horizontal axis. A perfect model, where observed = predicted, is shown by the line. HAL = Help with the Assessment of Adenopathy in Lung cancer model.

Figure 3

Receiver operating characteristic curves of the HOMER model in the combined validation cohort. The figure plots the area under the curve (AUC) for A, N1 or higher (vs N0) disease (AUC = 0.837) and B, N2 or N3 (vs N0 or N1) disease (AUC = 0.876) for all patients that met inclusion criteria from all non-MDACC institutions (n = 1,244), and for C, N1 or higher (vs N0) disease (AUC = 0.837) and D, N2 or N3 (vs N0 or N1) disease (AUC = 0.878) for patients that met inclusion criteria from non-MDACC institutions that had more than 50 eligible patients (n = 1,146). HOMER = Help with Oncologic Mediastinal Evaluation for Radiation; MDACC = MD Anderson Cancer Center.

Figure 4

Observed vs predicted plots for the HOMER model in the combined validation cohort. The figure plots the probability of N1 or higher (vs N0) disease for A and the probability (B) of N2 or N3 (vs N0 or N1) disease by decile of expected risk for all patients that met inclusion criteria from all non-MDACC institutions (n = 1,244) and the probability of (C) N1 or higher (vs N0) disease and the probability of (D) N2 or N3 (vs N0 or N1) disease by decile of expected risk for patients that met inclusion criteria from non-MDACC institutions that had more than 50 eligible patients (n = 1,146). The observed probability for each decile is on the vertical axis; the predicted probability for each decile is on the horizontal axis. A perfect model, where observed = predicted, is shown by the line. HOMER = Help with Oncologic Mediastinal Evaluation for Radiation; MDACC = MD Anderson Cancer Center.