Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Aug 13:11:1429259.
doi: 10.3389/fnut.2024.1429259. eCollection 2024.

Zero-shot evaluation of ChatGPT for food named-entity recognition and linking

Matevž Ogrinc  1   2 , Barbara Koroušić Seljak  2 , Tome Eftimov  2
Affiliations

Zero-shot evaluation of ChatGPT for food named-entity recognition and linking

Matevž Ogrinc et al. Front Nutr. .

Abstract

Introduction: Recognizing and extracting key information from textual data plays an important role in intelligent systems by maintaining up-to-date knowledge, reinforcing informed decision-making, question-answering, and more. It is especially apparent in the food domain, where critical information guides the decisions of nutritionists and clinicians. The information extraction process involves two natural language processing tasks named entity recognition-NER and named entity linking-NEL. With the emergence of large language models (LLMs), especially ChatGPT, many areas began incorporating its knowledge to reduce workloads or simplify tasks. In the field of food, however, we noticed an opportunity to involve ChatGPT in NER and NEL.

Methods: To assess ChatGPT's capabilities, we have evaluated its two versions, ChatGPT-3.5 and ChatGPT-4, focusing on their performance across both NER and NEL tasks, emphasizing food-related data. To benchmark our results in the food domain, we also investigated its capabilities in a more broadly investigated biomedical domain. By evaluating its zero-shot capabilities, we were able to ascertain the strengths and weaknesses of the two versions of ChatGPT.

Results: Despite being able to show promising results in NER compared to other models. When tasked with linking entities to their identifiers from semantic models ChatGPT's effectiveness falls drastically.

Discussion: While the integration of ChatGPT holds potential across various fields, it is crucial to approach its use with caution, particularly in relying on its responses for critical decisions in food and bio-medicine.

Keywords: ChatGPT; food data; named-entity linking; named-entity recognition; natural language processing.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Figures

Figure 1
Figure 1
Food NER example from a recipe text.
Figure 2
Figure 2
NEL example, where “cream cheese” is linked to the SNOMED-CT ontology.
Figure 3
Figure 3
Pipeline for ChatGPT evaluation.
Figure 4
Figure 4
Comparison between ChatGPT-3.5 and ChatGPT-4 in finding correct food entities.
Figure 5
Figure 5
Comparison between ChatGPT-3.5 and ChatGPT-4 in finding biomedical entities.
Figure 6
Figure 6
Comparison between ChatGPT-3.5 and ChatGPT-4 in linking biomedical entities to identifiers.
See this image and copyright information in PMC

References

    1. Mozaffarian D, Aspry KE, Garfield K, Kris-Etherton P, Seligman H, Velarde GP, et al. “Food Is Medicine” strategies for nutrition security and cardiometabolic health equity. J Am Coll Cardiol. (2024) 83:843–64. 10.1016/j.jacc.2023.12.023 - DOI - PubMed
    1. Tanna N. The impact of dietary guidelines for americans on dietary intake and obesity rates; 2024. In: Copyright - Database Copyright ProQuest LLC; ProQuest Does Not Claim Copyright in the Individual Underlying Works. Available at: https://www.proquest.com/dissertations-theses/impact-dietary-guidelines-... (accessed February 12, 2024).
    1. Nadeau D, Sekine S. A survey of named entity recognition and classification. Lingvisticae Investigationes. (2007) 30:3–26. 10.1075/li.30.1.03nad - DOI
    1. Shen W, Wang J, Han J. Entity linking with a knowledge base: issues, techniques, and solutions. IEEE Trans Knowl Data Eng. (2015) 27:443–60. 10.1109/TKDE.2014.2327028 - DOI
    1. Zhou X, Zhang X, Hu X. MaxMatcher: Biological concept extraction using approximate dictionary lookup. In: Pacific RIM International Conference on Artificial Intelligence. Cham: Springer; (2006). p. 1145–1149.

LinkOut - more resources