Large Language Models as Decision-Making Tools in Oncology: Comparing Artificial Intelligence Suggestions and Expert Recommendations

Abstract

Purpose: To determine the accuracy of large language models (LLMs) in generating appropriate treatment options for patients with early breast cancer on the basis of their medical records.

Materials and methods: Retrospective study using anonymized medical records of patients with BC presented during multidisciplinary team meetings (MDTs) between January and April 2024. Three generalist artificial intelligence models (Claude3-Opus, GPT4-Turbo, and LLaMa3-70B) were used to generate treatment suggestions, which were compared with experts' decisions. The primary outcome was the rate of appropriate suggestions from the LLMs, compared with the reference experts' decisions. The secondary outcome was the LLMs' performances (F1 score and specificity) in generating appropriate suggestions for each treatment category.

Results: The rates of appropriate suggestions were 86.6% (97/112), 85.7% (96/112), and 75.0% (84/112) for Claude3-Opus, GPT4-Turbo, and LLaMa3-70B, respectively. No significant difference was found between Claude3-Opus and GPT4-Turbo (P = .85), but both tended to perform better than LLaMa3-70B (P = .027 and P = .043, respectively). LLMs showed high accuracy for adjuvant endocrine therapy and targeted therapy indications. However, they tended to overestimate the need for adjuvant radiotherapy and had variable performances in suggesting adjuvant chemotherapy and genomic tests.

Conclusion: LLMs, particularly Claude3-Opus and GPT4-Turbo, demonstrated promising accuracy in suggesting appropriate adjuvant treatments for patients with early BC on the basis of their medical records. Although LLMs showed limitations in validating surgery and indicating genomic tests, their performance in other treatment modalities highlights their potential to automate and augment decision making during MDTs. Further studies with fine-tuned LLMs and a prospective design are needed to demonstrate their utility in clinical practice.

FIG 1.

Illustration of the automation workflow to process medical records into LLMs. Each selected case from our databases represents an input into LLMs, which output suggestions about the treatment to be compared with decisions emitted by experts. The task was enunciated with the following prompt (translated from French): “You are an expert in oncology, particularly in breast cancer. I need you to review the attached medical file of a patient who needs to be treated. Based on your knowledge and skills in oncology, I want you to decide on the recommended complementary treatments. The first step is to tell me if the surgery is complete and validated by saying ‘validated surgery.’ If the surgery is validated, the second step is to list all the additional treatments indicated, including chemotherapy, anti–HER2-targeted therapy, endocrine therapy and radiotherapy.” HER2, human epidermal growth factor receptor 2; LLM, large language models.

FIG 2.

Repartition of the suggested treatments according to the three LLMs. (A) The number of suggestions that were concordant, acceptable, or unacceptable compared with the decisions emitted by human experts. (B) The rates of appropriate suggestions (which are the sum of concordant and acceptable) that represented our primary end point. The rates of appropriate suggestions appeared higher with Claude3-Opus and GPT4-Turbo in comparison with LLaMa3-70B. LLMs, large language models.

FIG 3.

Performances of the LLMs to appropriately suggest each treatment modality. (A) The F1-score and (B) specificity of the three LLMs according to each treatment modality. F1-score (on the basis of recall and precision) reflects the capacity to correctly identify patients requiring a specific treatment, whereas specificity reflects the capacity to correctly identify patients not indicated for a specific treatment. All three models were perfectly accurate for indications of adjuvant targeted therapy and endocrine therapy. Differences between the models were more visible for chemotherapy and genomic test indications, and for radiotherapy to a lesser extent. LLM, large language models.

FIG A1.

Recall and precision of the LLMs. This figure shows the recall, also known as sensitivity (above) and precision (below) of each model. These two metrics are including in the the F1-score according to the following formula: (2 x recall x precision)/(recall + precision). LLMs, large language models.