Homologous point transformer for multi-modality prostate image registration

Abstract

Registration is the process of transforming images so they are aligned in the same coordinate space. In the medical field, image registration is often used to align multi-modal or multi-parametric images of the same organ. A uniquely challenging subset of medical image registration is cross-modality registration-the task of aligning images captured with different scanning methodologies. In this study, we present a transformer-based deep learning pipeline for performing cross-modality, radiology-pathology image registration for human prostate samples. While existing solutions for multi-modality prostate image registration focus on the prediction of transform parameters, our pipeline predicts a set of homologous points on the two image modalities. The homologous point registration pipeline achieves better average control point deviation than the current state-of-the-art automatic registration pipeline. It reaches this accuracy without requiring masked MR images which may enable this approach to achieve similar results in other organ systems and for partial tissue samples.

Keywords: Control points; Deep learning; Medical imaging; Registration; Transformer.

Figure 1. Demonstration of the image processing pipeline.

Input to pipeline requires single slice histology image (moving) and corresponding 2-D MR slice (fixed) without masking. (Left) Solving for high information histology landmarks using a scale invariant feature transform (SIFT) algorithm and enforcing a specified point density. (Middle) Three head network design that accepts MR image, histology image, and proposed histology control points which then solves for corresponding MR homologous points. (Right) With homologous points solved in both domains, a homologous point registration algorithm (Thin-plate Spline) is applied and histology image is warped into MR space.

Figure 2. Network overview.

(A) Full homologous point prediction network. The network has three input heads; M—Grayscale whole mount histology slide (512 × 512), F—Grayscale T2—weighted MR image (512 × 512), X—Histology landmarks. The output of the network is the set of homologous points in MR space—Y. (B) Histology and MR images are divided into patches, prior to linear embedding (MR image shown). The number of patches is a function of the number of points found in the histology landmark generation step. The transformer then uses the linear patch embeddings to compare histology and MR images to find similar patches and propose homologous points. (C) Locality-preserving transformer encoder block that uses convolution as opposed to the commonly-used global multi-head mechanism.

Figure 3. Evaluation metrics.

(A) Example control point deviation. Average control point deviation is the mean length of the red lines between the ground truth points (blue) and the warped landmarks (orange). (B) Example Dice coefficient. M_A is red, M_B is blue, M_A $\cap$M_B is white.

Figure 4. Softmax output visualization for a single point.

Left—Original histology with single landmark (blue). Center—Corresponding MR slice with predicted point probability heatmap generated from the network. The predicted point probability is used to generate the estimated location of the originally placed histology landmark in MR space. Right—Corresponding MR slice with blue predicted point vs. yellow ground truth point.

Figure 5. Comparison of registration results.

Method evaluations from left to right; Human Ground Truth, ProsRegNet—Paper, ProsRegNet—Tuned, and Homologous Point Pipeline. Top— visualization of the Dice coefficient. Blue indicates the ground truth prostate mask in MR space, red indicates prostate mask of histology post registration, and white denotes overlap between the two masks. Top middle—Dice coefficients with individual prostate slides shown in color, black dotted line represents average Dice coefficient for each method. Bottom middle—visualization of the control point displacement between transformed histology space ground truth and MR space ground truth. Yellow dots mark MR point ground truth, blue dots mark transformed histology landmarks, red lines indicate error. Bottom—Mean control point deviation with individual prostate slides shown in color, black dotted line represents average control point deviation for each method. p-value < 0.005 marked with ‘*’.