In vitro machine learning-based CAR T immunological synapse quality measurements correlate with patient clinical outcomes

Abstract

The human immune system consists of a highly intelligent network of billions of independent, self-organized cells that interact with each other. Machine learning (ML) is an artificial intelligence (AI) tool that automatically processes huge amounts of image data. Immunotherapies have revolutionized the treatment of blood cancer. Specifically, one such therapy involves engineering immune cells to express chimeric antigen receptors (CAR), which combine tumor antigen specificity with immune cell activation in a single receptor. To improve their efficacy and expand their applicability to solid tumors, scientists optimize different CARs with different modifications. However, predicting and ranking the efficacy of different "off-the-shelf" immune products (e.g., CAR or Bispecific T-cell Engager [BiTE]) and selection of clinical responders are challenging in clinical practice. Meanwhile, identifying the optimal CAR construct for a researcher to further develop a potential clinical application is limited by the current, time-consuming, costly, and labor-intensive conventional tools used to evaluate efficacy. Particularly, more than 30 years of immunological synapse (IS) research data demonstrate that T cell efficacy is not only controlled by the specificity and avidity of the tumor antigen and T cell interaction, but also it depends on a collective process, involving multiple adhesion and regulatory molecules, as well as tumor microenvironment, spatially and temporally organized at the IS formed by cytotoxic T lymphocytes (CTL) and natural killer (NK) cells. The optimal function of cytotoxic lymphocytes (including CTL and NK) depends on IS quality. Recognizing the inadequacy of conventional tools and the importance of IS in immune cell functions, we investigate a new strategy for assessing CAR-T efficacy by quantifying CAR IS quality using the glass-support planar lipid bilayer system combined with ML-based data analysis. Previous studies in our group show that CAR-T IS quality correlates with antitumor activities in vitro and in vivo. However, current manually quantified IS quality data analysis is time-consuming and labor-intensive with low accuracy, reproducibility, and repeatability. In this study, we develop a novel ML-based method to quantify thousands of CAR cell IS images with enhanced accuracy and speed. Specifically, we used artificial neural networks (ANN) to incorporate object detection into segmentation. The proposed ANN model extracts the most useful information to differentiate different IS datasets. The network output is flexible and produces bounding boxes, instance segmentation, contour outlines (borders), intensities of the borders, and segmentations without borders. Based on requirements, one or a combination of this information is used in statistical analysis. The ML-based automated algorithm quantified CAR-T IS data correlates with the clinical responder and non-responder treated with Kappa-CAR-T cells directly from patients. The results suggest that CAR cell IS quality can be used as a potential composite biomarker and correlates with antitumor activities in patients, which is sufficiently discriminative to further test the CAR IS quality as a clinical biomarker to predict response to CAR immunotherapy in cancer. For translational research, the method developed here can also provide guidelines for designing and optimizing numerous CAR constructs for potential clinical development. Trial Registration: ClinicalTrials.gov NCT00881920.

Fig 1. Comparable CAR expressions between patient #3 and patient #4.

PBMCs from patients #3 and #4 were transduced with the kappa-CAR retrovirus, respectively. The ratio and expression (MFI) of CD8 and CD4 subsets are calculated. (A) Flow cytometry analysis of CD8 and CD4 positive population from patients #3 and #4. (B) The ratio of CD3 and CAR positive subsets is calculated. (C) Different subsets of CD3⁺ T cells and viability are summarized. Data are pooled from at least two independent experiments.

Fig 2. The model shows the process of perforin and pZeta cluster formation, and accumulation of F-actin formation after the initial contact of the CAR-T and planar lipid bilayer.

(A) At the initial contact of the CAR with the tumor antigen, micro clusters are formed around the receptor, and the cell starts to spread. (B) The cell spread, and multiple microclusters form. (C) After the cell spread, F-actin polymerizes at the cell periphery. The perforin and pZeta are transported toward the cell center along with F-actin. (D) Perforin and pZeta populate the actin-sparse center and form a cluster. In the experiment, we labeled the different substances with different colors, and different channels of images were obtained using different lasers. We use six single-cell samples in five channels using the best Z position.

Fig 3. The overall model of instance segmentation for CAR-T cells using multi-scale cell instance segmentation.

(A) Demonstrates the training phase. In this phase, CAR-T IS images are used for training sets. (B) Shows the model in the evaluation phase. In this phase, each sample has five channels, of which four of them are applicable for evaluation. Channel 3 is used to select the best Z slide, and Channel 1 provides the best possible representation of the CAR-T IS. From Channel 1, the network produces bounding boxes, instance segmentation, and contours. The generated masks and contours are applied on all channels for statistical analysis.

Fig 4. Comparison of generated instance segmentation masks in the evaluation phase with their ground truths.

We applied colormaps ’Magenta’, ’Green’, and ’Yellow’ for better representation of the images. Different shades are used to separate the cells from each other. Four different zooming areas are selected for analysis. The images with the same number point to the same zooming area.

Fig 5. The comparison of the model’s loss with different sets of training data.

(A) Represents a test image sample in the evaluation phase using the defined trained networks. In (B), four different zoomed areas are selected for analysis. Images with similar numbers point to the same boxes in (A). These images present the effect of having access to more training data and its role in removing discrepancies. (C) Shows the training loss, and (D) Shows the validation loss from 0 to 100 training iterations with 100% of the training data. As expected, the training loss shows a more predictable pattern than validation loss.

Fig 6. The total intensity in 4 channels.

F-actin at row 1 (channel 1), perforin at row 2 (channel 2), tumor antigen at row 3 (channel 3), pZeta at row 4 (channel 4). (A) Shows one sample for each patient. The left side is for patient #3 and the right side for patient #4. In these images, the regions that do not belong to any predicted masks from ANNs are removed. Auto-contrast makes cells visible to human eyes (they do not affect real analysis). The (B), (C), (D), and (E) show the total intensity distribution and cumulative probability of two patients using fully trained networks across all channels for all counted cells from the evaluation phase. The figures also show the mean, variance, and the number of cells detected for each channel separately.

Fig 7. Successful image data extraction in a python environment.

(A) Is a sample of 11 Z slides with five channels: F-actin at row 1 (channel 1), perforin at row 2 (channel 2), tumor antigen at row 3 (channel 3), pZeta at row 4 (channel 4) and, the DIC of the cells at row 5 (channel 5). To have clear representations of the cells in the figure, colormap filters are added to the original grayscale images: F-actin received ’RdGy_r’ colormap, perforin received ’PRGn_r’ colormap, tumor antigen received ’RdBu_r’ colormap, pZeta received ’PuOr_r’ colormap and, the DIC received ’binary’ colormap. The colormaps [61] are only used for representation purposes and do not affect the evaluation of the IS. (B) Plots the mean intensity values for grayscale images through Z slides for all channels.

Fig 8. The outputs of multi-scale cell instance segmentation.

For illustration, we use three different images in three different rows. The first row is for a sparsely populated image, the second row is for a moderately populated image, and the third row is for a highly populated image. The framework contains two modules: (a) bounding box detection module and (b) individual cell segmentation module. The bounding box detection outputs the bounding boxes over each detected cell. The bounding box determines an object by indicating the top-left, top-right, bottom-left, bottom-right, and center points, respectively. The bounding boxes are used to create patches of cells, used for instance segmentation. The instance segmentation masks can be used to create borders (contours) and segmentations without borders, which are inside the areas of the segmented objects.