Developing a named entity framework for thyroid cancer staging and risk level classification using large language models

Abstract

We developed a named entity (NE) framework for information extraction from semi-structured clinical notes retrieved from The Cancer Genome Atlas-Thyroid Cancer (TCGA-THCA) database and examined Large Language Models (LLMs) strategies to classify the 8^th edition of American Joint Committee on Cancer (AJCC) staging and American Thyroid Association (ATA) risk category for patients with well-differentiated thyroid cancer. The NE framework consisted of annotation guidelines development, ground truth labelling, prompting approaches, and evaluation codes. Four LLMs (Mistral-7B-Instruct, Llama-3.1-8B-Instruct, Gemma-2-9B-Instruct, and Qwen2.5-7B-Instruct) were offline utilised for information extraction, comparing with expert-curated ground truth. Our framework was developed using 50 TCGA-THCA pathology notes. 289 TCGA-THCA notes and 35 pseudo-clinical cases were used for validation. Taking an ensemble-like majority-vote strategy achieved satisfactory performance for AJCC and ATA in both development and validation sets. Our framework and ensemble classifier optimised efficiency and accuracy of classifying stage and risk category in thyroid cancer patients.

Fig. 1. Flowchart of patient selection process.

Flowchart depicting patient selection and the data source used as development set and validation set. Cancer stages and ATA risks of all TCGA-THCA patients and pseudo cases were verified by endocrine surgeons. A pseudo case of non-invasive follicular thyroid neoplasm with papillary like nuclear features is not grade with AJCC staging and ATA risk.

Fig. 2. Flow of data extraction using LLMs and classifying ATA risk and AJCC staging from the LLM output.

Schematic diagram depicting the flow of data extraction using LLMs and the utilization of self-developed Microsoft Excel template for data cleaning and classification.

Fig. 3. Heatmap of performance of Large Language Models on classification of ATA risks and AJCC staging in 50 TCGA pathology reports for NE framework development.

LLMs with various prompting strategies attained satisfactory performance in NE framework development. a Performance on ATA risk classification with F1-scores 88.0–100.0%. b Performance on AJCC staging with F1-scores of 90.3–100.0%.

Fig. 4. Heatmap of performance of ensemble classifiers on classification of ATA risks and AJCC staging in the development and validation sets.

Ensemble classifiers attained satisfactory performance on the two datasets. a Performance on ATA risk classification with F1-scores at least 88.5%. b Performance on AJCC staging with F1-scores of at least 90.4%.

Fig. 5. Heatmap of performance of Large Language Models on classification of ATA risks and AJCC staging in 289 TCGA pathology reports for validation.

LLMs with various prompting strategies attained satisfactory performance in 289 TCGA pathology reports for validation. a Performance on ATA risk classification. with F1-scores 88.5–96.5%. b Performance on AJCC staging with F1-scores 94.2–99.7%.

Fig. 6. Heatmap of performance of Large Language Models on classification of ATA risks and AJCC staging in 35 pseudo cases for validation.

The performance of LLMs various in different approach and in individual LLM in the 35 pseudo cases for validation. a Performance on ATA risk classification. Mistral-7B-Instruct-v0.3 outperformed other LLMs with F1-score of 94.3%. b Performance on AJCC staging. Llama-3.1-8B-Instruct outperformed other LLMs with F1-score of 97.5%.