InfectA-Chat, an Arabic Large Language Model for Infectious Diseases: Comparative Analysis

Abstract

Background: Infectious diseases have consistently been a significant concern in public health, requiring proactive measures to safeguard societal well-being. In this regard, regular monitoring activities play a crucial role in mitigating the adverse effects of diseases on society. To monitor disease trends, various organizations, such as the World Health Organization (WHO) and the European Centre for Disease Prevention and Control (ECDC), collect diverse surveillance data and make them publicly accessible. However, these platforms primarily present surveillance data in English, which creates language barriers for non-English-speaking individuals and global public health efforts to accurately observe disease trends. This challenge is particularly noticeable in regions such as the Middle East, where specific infectious diseases, such as Middle East respiratory syndrome coronavirus (MERS-CoV), have seen a dramatic increase. For such regions, it is essential to develop tools that can overcome language barriers and reach more individuals to alleviate the negative impacts of these diseases.

Objective: This study aims to address these issues; therefore, we propose InfectA-Chat, a cutting-edge large language model (LLM) specifically designed for the Arabic language but also incorporating English for question and answer (Q&A) tasks. InfectA-Chat leverages its deep understanding of the language to provide users with information on the latest trends in infectious diseases based on their queries.

Methods: This comprehensive study was achieved by instruction tuning the AceGPT-7B and AceGPT-7B-Chat models on a Q&A task, using a dataset of 55,400 Arabic and English domain-specific instruction-following data. The performance of these fine-tuned models was evaluated using 2770 domain-specific Arabic and English instruction-following data, using the GPT-4 evaluation method. A comparative analysis was then performed against Arabic LLMs and state-of-the-art models, including AceGPT-13B-Chat, Jais-13B-Chat, Gemini, GPT-3.5, and GPT-4. Furthermore, to ensure the model had access to the latest information on infectious diseases by regularly updating the data without additional fine-tuning, we used the retrieval-augmented generation (RAG) method.

Results: InfectA-Chat demonstrated good performance in answering questions about infectious diseases by the GPT-4 evaluation method. Our comparative analysis revealed that it outperforms the AceGPT-7B-Chat and InfectA-Chat (based on AceGPT-7B) models by a margin of 43.52%. It also surpassed other Arabic LLMs such as AceGPT-13B-Chat and Jais-13B-Chat by 48.61%. Among the state-of-the-art models, InfectA-Chat achieved a leading performance of 23.78%, competing closely with the GPT-4 model. Furthermore, the RAG method in InfectA-Chat significantly improved document retrieval accuracy. Notably, RAG retrieved more accurate documents based on queries when the top-k parameter value was increased.

Conclusions: Our findings highlight the shortcomings of general Arabic LLMs in providing up-to-date information about infectious diseases. With this study, we aim to empower individuals and public health efforts by offering a bilingual Q&A system for infectious disease monitoring.

Keywords: AceGPT; Arabic large language models; infectious disease monitoring; large language model; multilingual large language model; public health.

Figure 1

Modeling of InfectA-Chat. RAG: retrieval-augmented generation.

Figure 2

The overall pipeline of data collection and preprocessing. Q&A: question and answer.

Figure 3

The overall pipeline of supervised fine-tuning (instruction tuning).

Figure 4

The overall pipeline of Retrieval-Augmented Generation.

Figure 5

Training progress of AceGPT-7B–based InfectA-Chat (training loss, validation loss, and learning rate). SFT: supervised fine-tuning.

Figure 6

Training progress of AceGPT-7B-Chat–based InfectA-Chat (training loss, validation loss, and learning rate). SFT: supervised fine-tuning.

Figure 7

Qualitative performance comparison for AceGPT-7B-Chat, AceGPT-7B–based InfectA-Chat (ours), and AceGPT-7B-Chat–based InfectA-Chat (ours) by GPT-4 evaluation.

Figure 8

Qualitative performance comparison for AceGPT-13B-Chat, Jais-13B-Chat, and AceGPT-7B-Chat–based InfectA-Chat (ours) by GPT-4 evaluation.

Figure 9

Qualitative performance comparison for InfectA-Chat (ours), GPT-4, GPT-3.5, and Gemini.

Figure 10

Qualitative performance results for InfectA-Chat on the recent dataset (March 20, 2024, to June 20, 2024) by GPT-4 evaluation.

Figure 11

Accuracy of retriever based on top-k parameter in Retrieval-Augmented Generation pipeline.