Aligning knowledge concepts to whole slide images for precise histopathology image analysis

Abstract

Due to the large size and lack of fine-grained annotation, Whole Slide Images (WSIs) analysis is commonly approached as a Multiple Instance Learning (MIL) problem. However, previous studies only learn from training data, posing a stark contrast to how human clinicians teach each other and reason about histopathologic entities and factors. Here, we present a novel knowledge concept-based MIL framework, named ConcepPath, to fill this gap. Specifically, ConcepPath utilizes GPT-4 to induce reliable disease-specific human expert concepts from medical literature and incorporate them with a group of purely learnable concepts to extract complementary knowledge from training data. In ConcepPath, WSIs are aligned to these linguistic knowledge concepts by utilizing the pathology vision-language model as the basic building component. In the application of lung cancer subtyping, breast cancer HER2 scoring, and gastric cancer immunotherapy-sensitive subtyping tasks, ConcepPath significantly outperformed previous SOTA methods, which lacked the guidance of human expert knowledge.

Fig. 1. Overview of ConcepPath framework.

a In real clinical processes, pathologists apply their expert knowledge to reason about histopathologic entities and factors to make a diagnosis. b ConcepPath utilizes a large language model like GPT-4 to induce expert concepts related to diagnosis from medical literature and integrate this knowledge into an automated WSI analysis pipeline through the CLIP-based pathology vision-language foundation model. c Dataset Characteristics of the evaluated tasks. d Left: An illustration of how the CLIP-based pathology foundation model performs patch prediction with class prompts. Right: The pipeline of how the previous CLIP-based pathology foundation model performs slide-level classification via a top-k pooling of patch predictions. e Left: An illustration of how ConcepPath decomposes a specific complex WSI analysis task into multiple subtasks of scoring patch-level concepts/attributes. Right: The pipeline of how ConcepPath conducts slide-level classification. Unlike the previous CLIP-based pathology foundation models' mechanism, ConcepPath leverages human prior knowledge and fully exploits the power of the CLIP-based pathology foundation model by scoring a group of expert concepts induced by GPT-4 from related medical literature and extracting complementary knowledge from training data via scoring a group of learnable data-driven concepts. The final prediction is produced with a two-stage aggregation mechanism with the above concepts.

Fig. 2. Performance and expert concept generation comparison of ConcepPath.

a Radar charts depicting the average AUC(Left) and ACC(Right) for the five WSI analysis tasks in the five-fold cross-validation experiment conducted on NSCLC, BRCA, and STAD datasets. “Average” denotes the average performance among all five tasks. “TOP*” represents the higher performance in the author’s implementation and our implementation of TOP. ConcepPath demonstrated more accurate predictions on all five tasks since it successfully incorporates human expert prior knowledge and data-driven knowledge learned from the training data. b, c A misleading concept generated by directly querying GPT-4 (denoted as “Generated”), which our induction-based method (denoted as “Induced”) successfully rectifies for the gastric immunotherapy-sensitive subtyping task. Specifically, the concept “signet-ring cells” was found in the category of Epstein–Barr virus (EBV) positive subtype in the generated concepts; however, this morphology is more commonly linked to the Genomically Stable (GS) subtype, where mutations in CDH1 and RHO genes play a pivotal role. d A histogram representing the AUC comparison of different expert concept generation strategies. The y-axis is the AUC(%) and the x-axis is the WSI analysis tasks and their average performance. “Induced” concepts demonstrated better performance among all tasks, especially for the more challenging immunotherapy-sensitive subtyping tasks on the STAD dataset, highlighting the importance of inducing concepts from professional materials for complex WSI analysis tasks.

Fig. 3. Investigation of proposed components in ConcepPath.

a Line plots for investigating new data-driven concept learning, the y-axis is the AUC(%) and the x-axis is the number of learned concepts used for each class, where 0 means only using human expert concepts. Integrating data-driven knowledge learned from training data improves overall performance, and for more challenging and inadequately researched diagnostic tasks, a greater number of learned concepts are required. The performance decline when the number of learned concepts was large suggests a potential trade-off between prior expert knowledge and learned knowledge. b A histogram representing investigations on second-stage bag-level concept-guided aggregation and slide adapters, the y-axis is the AUC(%), and “w/o Bag-level guidance” refers to using average pooling aggregation. Both modules contributed to the improved performance. c A histogram representing the comparison of using different CLIP-based vision-language models as ConcepPath’s basic component for aligning histopathology images and concept knowledge, and the y-axis is the AUC(%). Obvious performance increase can be observed by using pathology vision-language models, and ConcepPath could benefit from more accurate alignment if the pathology vision-language were trained on larger datasets.

Fig. 4. Visualizations of ConcepPath and baseline method.

Instance-level expert concept similarity maps. The slides are accurately identified as the lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD) subtype, respectively. In comparison to the CLAM attention map, the similarity maps of various instance-level concepts provide a more precise focus on the tumor in the green box highlighted area.