Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Aug 1;34(8):1302-1304.
doi: 10.1681/ASN.0000000000000166. Epub 2023 May 31.

Retrieve, Summarize, and Verify: How Will ChatGPT Affect Information Seeking from the Medical Literature?

Qiao Jin  1 , Robert Leaman, Zhiyong Lu
Affiliations

Retrieve, Summarize, and Verify: How Will ChatGPT Affect Information Seeking from the Medical Literature?

Qiao Jin et al. J Am Soc Nephrol. .
No abstract available

PubMed Disclaimer

Conflict of interest statement

All authors have nothing to disclose.

Figures

Figure 1
Figure 1
Seeking information related to COVID-19 from ChatGPT. (A) ChatGPT-3.5 fabricates the references for its claims. (B) When prompted with the title and abstract of relevant articles, ChatGPT-3.5 can summarize the aspects requested, including some valid in-line references. However, many details are inaccurate: Related mechanisms are not correctly grouped, contradicting evidence is not mentioned, some citations are missing, and a listed mechanism is not specific. Note that the complete response mentioned only three of five articles. (C) ChatGPT-4 acknowledges its drawback and refuses to cite source articles in the response, which makes the verification more difficult. (D) ChatGPT-4 with the web browsing plug-in summarizes information from the search engine, but it only finds two relevant articles and fails to systematically answer the question. Responses were collected from GPT-3.5–based ChatGPT on March 14, 2023, and GPT-4–based ChatGPT on May 15, 2023.
See this image and copyright information in PMC

Comment in

References

    1. OpenAI: ChatGPT. Optimizing Language Models for Dialogue. Accessed November 30. https://openai.com/blog/chatgpt/
    1. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2:e0000198. doi:10.1371/journal.pdig.0000198 - DOI - PMC - PubMed
    1. Patel SB, Lam K, ChatGPT. The future of discharge summaries? Lancet Digit Health. 2023;5(3):e107–e108. doi:10.1016/S2589-7500(23)00021-3 - DOI - PubMed
    1. Moor M, Banerjee O, Abad ZSH, Krumholz HM, Leskovec J, Topol EJ. Foundation models for generalist medical artificial intelligence. Nature. 2023;616(7956):259–265. doi:10.1038/s41586-023-05881-4 - DOI - PubMed
    1. Singhal K Azizi S Tu T, et al. . Large language models encode clinical knowledge [published online ahead of print December 26, 2022]. arXiv. 2022:221213138. doi:10.48550/arXiv.2212.13138 - DOI

Publication types

MeSH terms

LinkOut - more resources