Simulate Scientific Reasoning with Multiple Large Language Models: An Application to Alzheimer's Disease Combinatorial Therapy

Abstract

Motivation: This study aims to develop an AI-driven framework that leverages large language models (LLMs) to simulate scientific reasoning and peer review to predict efficacious combinatorial therapy when data-driven prediction is infeasible.

Results: Our proposed framework achieved a significantly higher accuracy (0.74) than traditional knowledge-based prediction (0.52). An ablation study highlighted the importance of high quality few-shot examples, external knowledge integration, self-consistency, and review within the framework. The external validation with private experimental data yielded an accuracy of 0.82, further confirming the framework's ability to generate high-quality hypotheses in biological inference tasks. Our framework offers an automated knowledge-driven hypothesis generation approach when data-driven prediction is not a viable option.

Availability and implementation: Our source code and data are available at https://github.com/QidiXu96/Coated-LLM.

Figure 1.

Study overview. Coated-LLM is a structured framework that mimics human scientific reasoning and peer review processes to generate hypotheses on efficacious combinatorial therapy. It consists of three stages: (i) Warm-up phase, where Researcher uses external biological knowledge to practice scientific inference and keep correct predictions as learning examples. (ii) Inference phase, where Researcher inferences the new combination using its top five similar questions from learning examples and gets the consistency prediction. (iii) Revision phase, where multiple Reviewers provide feedback and Moderator integrates consistency prediction from Researcher and feedback from Reviewer to generate the final consensus prediction.

Figure 2.. Distribution of drug combinations and efficacy in literature

a. Data collection from literature. The process began with an initial pool of articles from the AlzPED, followed by additional searches conducted in PubMed. Articles were screened and excluded based on predefined criteria. The final selected literatures included articles that reported drug combinations with positive or negative efficacy. b. Top 5 frequent terms in therapeutic agents, animal models, and pathways. c. UMAP visualization of drug combinations and efficacy. Each drug combination is converted into a natural language question to generate embeddings with OpenAI’s text-embedding-ada-002 model. The UMAP projection, derived from these embeddings, reveals that the combination (AMD3100, L-Lactate, 3xTg), for example, is similar to combinations which have same animal model (e.g., ABT-107, Donepezil, 3xTg).

Figure 3.

Visual illustration of Coated-LLM components and additive contributions to the performance. Coated-LLM combines kNN-based five-shots dynamic learning example selection, external pathway knowledge, self-consistency (n=5), Reviewer, and Moderator.