A multi-view validation framework for LLM-generated knowledge graphs of chronic kidney disease
- PMID: 40810826
- DOI: 10.1007/s11548-025-03495-x
A multi-view validation framework for LLM-generated knowledge graphs of chronic kidney disease
Abstract
Purpose: The goal of our work is to develop a multi-view validation framework for evaluating LLM-generated knowledge graph (KG) triples. The proposed approach aims to address the lack of established validation procedure in the context of LLM-supported KG construction.
Methods: The proposed framework evaluates the LLM-generated triples across three dimensions: semantic plausibility, ontology-grounded type compatibility, and structural importance. We demonstrate the performance for GPT-4 generated concept-specific (e.g., for medications, diagnosis, procedures) triples in the context of chronic kidney disease (CKD).
Results: The proposed approach consistently achieves high-quality results across evaluated GPT-4 generated triples, strong semantic plausibility (semantic score mean: 0.79), excellent type compatibility (type score mean: 0.84), and high structural importance of entities within the CKD knowledge domain (ResourceRank mean: 0.94).
Conclusion: The validation framework offers a reliable and scalable method for evaluating quality and validity of LLM-generated triples across three views: semantic plausibility, type compatibility, and structural importance. The framework demonstrates robust performance in filtering high-quality triples and lays a strong foundation for fast and reliable medical KG construction and validation.
Keywords: Chronic Kidney Disease; Expert system; Healthcare; Knowledge graphs; Knowledge-driven modelling; LLMs; Model-guided medicine; Nephrology; Semantic evaluation; Triples; Validation; medical informatics.
© 2025. The Author(s).
Conflict of interest statement
Declarations. Conflict of interest: The authors declare no competing interests. Ethical statement: For MIMIC-IV, the collection of patient information and creation of the research resource was reviewed by the Institutional Review Board at the Beth Israel Deaconess Medical Center, who granted a waiver of informed consent and approved the data sharing initiative.
Similar articles
-
Ontology enrichment using a large language model: Applying lexical, semantic, and knowledge network-based similarity for concept placement.J Biomed Inform. 2025 Aug;168:104865. doi: 10.1016/j.jbi.2025.104865. Epub 2025 Jun 19. J Biomed Inform. 2025. PMID: 40543734
-
MedKA: A knowledge graph-augmented approach to improve factuality in medical Large Language Models.J Biomed Inform. 2025 Aug;168:104871. doi: 10.1016/j.jbi.2025.104871. Epub 2025 Jul 8. J Biomed Inform. 2025. PMID: 40639554
-
Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study.JMIR AI. 2025 Feb 24;4:e58670. doi: 10.2196/58670. JMIR AI. 2025. PMID: 39993309 Free PMC article.
-
Large Language Model Architectures in Health Care: Scoping Review of Research Perspectives.J Med Internet Res. 2025 Jun 19;27:e70315. doi: 10.2196/70315. J Med Internet Res. 2025. PMID: 40536801 Free PMC article.
-
Large Language Models and Empathy: Systematic Review.J Med Internet Res. 2024 Dec 11;26:e52597. doi: 10.2196/52597. J Med Internet Res. 2024. PMID: 39661968 Free PMC article.
References
-
- Zamini M, Reza H, Rabiei M (2022) A review of knowledge graph completion. Information 13(8):396
-
- Piovesan L, Molino G, Terenziani P (2014) An ontological knowledge and multiple abstraction level decision support system in healthcare. Decis Anal 1:1–24
-
- Gyrard A, Jaimini U, Gaur M, Shekharpour S, Thirunarayan K, Sheth A (2022) Reasoning over personalized healthcare knowledge graph: a case study of patients with allergies and symptoms. In: Semantic models in IoT and Ehealth applications. Academic Press, pp 199–225
-
- Choi E, Xu Z, Li Y, Dusenberry M, Flores G, Xue E, Dai A (2020) Learning the graphical structure of electronic health records with graph convolutional transformer. Proc AAAI Conf Artif Intell 34(01):606–613
Grants and funding
LinkOut - more resources
Full Text Sources