PMC-LLaMA: toward building open-source language models for medicine

Abstract

Objective: Recently, large language models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering (QA) situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this article, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA.

Materials and methods: We adapt a general-purpose LLM toward the medical domain, involving data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive domain-specific instruction fine-tuning, encompassing medical QA, rationale for reasoning, and conversational dialogues with 202M tokens.

Results: While evaluating various public medical QA benchmarks and manual rating, our lightweight PMC-LLaMA, which consists of only 13B parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, and datasets for instruction tuning will be released to the research community.

Discussion: Our contributions are 3-fold: (1) we build up an open-source LLM toward the medical domain. We believe the proposed PMC-LLaMA model can promote further development of foundation models in medicine, serving as a medical trainable basic generative language backbone; (2) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component, demonstrating how different training data and model scales affect medical LLMs; (3) we contribute a large-scale, comprehensive dataset for instruction tuning.

Conclusion: In this article, we systematically investigate the process of building up an open-source medical-specific LLM, PMC-LLaMA.

Keywords: ChatGPT; biomedical NLP; generative language models; large language models.

Figure 1.

In the left, we show the general comparison between our PMC-LLaMA with LLaMA-2 and GPT-3.5. On the right, we visually show the advantages of our model in model sizes. PMC-LLaMA is much smaller than the others.

Figure 2.

Distribution of medical textbooks categories. The box sizes denote the book numbers for different categories.

Figure 3.

The training pipeline of PMC-LLaMA. Our training flow can be separated into 2 parts, that is, data-centric knowledge injection and medical-specific instruction tuning. In knowledge injection, we collect 4.8M biomedical academic papers and 30K medical books for further injecting knowledge into LLaMA. In the instruction tuning stage, we mainly consider 3 aspects, medical conversation, medical rationale question answering, and knowledge graph, containing 202M tokens in total.

Figure 4.

Examples of 3 instruction prompting cases from PMC-LLaMA and GPT-3.5. (A) Compares their responses to the patient’s query, where PMC-LLaMA proposes more concrete suggestions. (B) Shows the probing of microbiology knowledge. PMC-LLaMA analyzes both correct and incorrect options, enhancing the comprehensiveness of the analysis. Example (C) examines the models’ grasp of Pharmacology, and they respond with roughly equivalent answers. The correct options are marked bold.