Automated classification of tertiary lymphoid structures in colorectal cancer using TLS-PAT artificial intelligence tool

Abstract

Colorectal cancer (CRC) ranks as the third most common and second deadliest cancer worldwide. The immune system, particularly tertiary lymphoid structures (TLS), significantly influences CRC progression and prognosis. TLS maturation, especially in the presence of germinal centers, correlates with improved patient outcomes; however, consistent and objective TLS assessment is hindered by varying histological definitions and limitations of traditional staining methods. This study involved 656 patients with colorectal adenocarcinoma from CHU Brest, France. We employed dual immunohistochemistry staining for CD21 and CD23 to classify TLS maturation stages in whole-slide images and implemented a fivefold cross-validation. Using ResNet50 and Vision Transformer models, we compared various aggregation methods, architectures, and pretraining techniques. Our automated system, TLS-PAT, achieved high accuracy (0.845) and robustness (kappa = 0.761) in classifying TLS maturation, particularly with the Vision Transformer pretrained on ImageNet using Max Confidence aggregation. This AI-driven approach offers a standardized method for automated TLS classification, complementing existing detection techniques. Our open-source tools are designed for easy integration with current methods, paving the way for further research in external datasets and other cancer types.

Keywords: Artificial intelligence; Colorectal cancer; Deep learning; Immunohistochemistry; QuPath; Tertiary lymphoid structures.

Fig. 1

Annotations Methodology of the Dataset and Strategic Dataset Design for Balanced TLS Maturation Stage Classification. HES and IHC slides (CD20/CD3/Ki67/CD31) and CD21/CD23) for the characterization of the different TLS maturation class: Aggregate (A), Non-GC TLS (B) and GC TLS (C). The whole dataset was split into training, validation and test sets (D), with an equivalent repartition of each TLS maturation class (E).

Fig. 2

Novel Automated Pipeline for TLS Maturation Classification. First the slide is scanned and TLS are detected and annotated by a pathologist in QuPath. Secondly, using a java script, we generated a low-resolution image of the whole slide with its tiled regions, and we extracted the tiles of each TLS. These tiles and their corresponding maturation class (csv file) are used as inputs for the best combination evaluation (model, pretraining, aggregation). As outputs, we obtained the same tiles with a color filter corresponding to its predicted class (csv file): Blue for Aggregate, Green for Non-GC and Red for GC. Tiles are then rearranged to allow a visualization of the results. Each tile colored with its predicted class, and the predicted class of the whole TLS (csv file), depending on the aggregation method chosen.

Fig. 3

Performance Comparison of ResNet50 and Vision Transformer (ViT) Models Using 5-Fold Cross-Validation. On the left, the accuracy results of the 5-Fold Cross-Validation with the Resnet50 model. On the right, the accuracy results of the 5-Fold Cross-Validation with the ViT large model.

Fig. 4

Normalized Confusion Matrices Comparing Custom and Max Confidence Aggregation Methods. The normalized confusion matrices using the Custom aggregation method with the Resnet-imagenet model (A), ViT-imagenet model (B) or the Vit-Uni model (C). The normalized confusion matrices using the Ax Confidence aggregation method with the Resnet-imagenet model (D), ViT-imagenet model (E) or the Vit-Uni model (F).

Fig. 5

Visual Insights into Model Performance and Error Patterns. Colored tiles are rearranged to visualize each entire TLS. Here are examples for each TLS maturation class, with the Resnet-imagenet model (A), ViT-imagenet model (B) or the Vit-UNI model (C). We observed some misclassification with the Resnet-imagenet model and Vit-Uni model, compared to the ViT-imagenet model (D).