Real-time liver tumor localization via a single x-ray projection using deep graph neural network-assisted biomechanical modeling

Abstract

Objective.Real-time imaging is highly desirable in image-guided radiotherapy, as it provides instantaneous knowledge of patients' anatomy and motion during treatments and enables online treatment adaptation to achieve the highest tumor targeting accuracy. Due to extremely limited acquisition time, only one or few x-ray projections can be acquired for real-time imaging, which poses a substantial challenge to localize the tumor from the scarce projections. For liver radiotherapy, such a challenge is further exacerbated by the diminished contrast between the tumor and the surrounding normal liver tissues. Here, we propose a framework combining graph neural network-based deep learning and biomechanical modeling to track liver tumor in real-time from a single onboard x-ray projection.Approach.Liver tumor tracking is achieved in two steps. First, a deep learning network is developed to predict the liver surface deformation using image features learned from the x-ray projection. Second, the intra-liver deformation is estimated through biomechanical modeling, using the liver surface deformation as the boundary condition to solve tumor motion by finite element analysis. The accuracy of the proposed framework was evaluated using a dataset of 10 patients with liver cancer.Main results.The results show accurate liver surface registration from the graph neural network-based deep learning model, which translates into accurate, fiducial-less liver tumor localization after biomechanical modeling (<1.2 (±1.2) mm average localization error).Significance.The method demonstrates its potentiality towards intra-treatment and real-time 3D liver tumor monitoring and localization. It could be applied to facilitate 4D dose accumulation, multi-leaf collimator tracking and real-time plan adaptation. The method can be adapted to other anatomical sites as well.

Keywords: biomechanical modeling; deep learning; graph neural network; liver; real-time tumor localization; x-ray.

Figure 1.

Workflow of the proposed method (MeshRegNet-Bio) to estimate liver tumor motion from a single real-time onboard x-ray projection. In step (a), a deep graph neural network (GNN)-based model was trained to predict the liver surface deformation vector field (DVF) from a single x-ray projection. In the subsequent step (b), a biomechanical model solves the intra-liver DVF using the liver surface DVF as the boundary condition for tumor localization. The liver segmentation and liver surface/volumetric mesh generation steps are offline and can be done at any point after the prior image is acquired, taking no onboard imaging time.

Figure 2.

Overview of the deep graph neural network (GNN)-driven architecture that estimates liver boundary motion from a single real-time onboard x-ray projection. The network consists of two subnetworks performing feature extraction and liver boundary DVF prediction separately. The first subnetwork uses ResNet-50 to extract image features from an x-ray projection. The extracted feature maps were pooled for each node of a liver surface mesh by the perceptual feature pooling layer, based on the projected node coordinates on the x-ray projection. The second subnetwork, referred to collectively as GNN in this work, consisting of three deformation modules, progressively estimates the liver boundary DVFs. A deformation module comprises of a graph convolutional network (GCN) block and a spatial transform layer. The GCN was trained to predict a liver boundary DVF based on the features extracted from the ResNet-50 subnetwork. A spatial transform layer deforms the prior liver reference mesh or the deformed liver surface mesh from the previous deformation module, using a GCN-predicted DVF.

Figure 3.

Projection geometry used in the perceptual feature pooling layer. The same cone-beam geometry as x-ray imaging was used for the feature pooling, where features were pooled from the ResNet-50 extracted 2D feature maps based on the coordinates of the projected surface mesh nodes.

Figure 4.

Graph convolutional network (GCN) block. The inputs contain pooled image features extracted from the ResNet-50 subnetwork (Figure 2), a surface DVF, and learned vertex features from the GCN in the previous deformation module (if any). The GCN consists of 14 graph convolutional layers that, except for the entrance and exiting layers, were organized in a residual learning architecture. The GCN yields a surface DVF and vertex features to feed into the subsequent deformation module (Figure 2). The inputs of the first GCN contains only image features and an initial surface DVF, which was set to zero. The image features were re-pooled for each subsequent GCN based on the deformed node coordinates (Figure 2). The rounded box in the middle represents a residual learning block containing two graph convolution layers with a shortcut connection, which iterates six times in our framework.

Figure 5.

Workflow of data augmentation to generate realistic respiration-correlated deformations. First, registrations between the prior reference-phase CT (0%) and the other respiratory phases were performed to obtain corresponding DVFs, using liver density-overridden four-dimensional (4D) CT images to enhance the DVF accuracy at liver boundaries. Second, the intra-liver DVFs were optimized through biomechanical modeling using the liver boundary DVFs as the boundary condition. Next, a patient-specific motion model was built by performing a principal component analysis (PCA) on the optimized DVFs to attain principal motion components and coefficients. Finally, realistic deformations were generated by randomly scaling the principal motion coefficients, which were used to propagate the reference-phase (0%) CT image to augmented deformation states.

Figure 6.

Examples of the regions of interest (ROIs) used in diaphragm tracking. The diaphragm was considered as a surrogate for respiratory motion and liver tumor motion. A ROI intersecting the diaphragm was selected for each patient for diaphragm tracking. The diaphragm displacement tracked by the ROI, in the superior-inferior direction, was used to represent the liver tumor center-of-mass motion. A relatively small ROI were used in this case for the 45° projection to avoid the intensity discontinuity artifacts in the x-ray projection.

Figure 7.

(First row) Liver surface overlays between the prior and ‘ground truth’ target meshes (left panel), and between the deep GNN-deformed and ‘ground truth’ target meshes (right panel), for two patient cases (a) and (b). The yellow meshes in both panels are the target meshes at the end-of-exhale phase with additional motion augmentation, and the red meshes correspond to the prior (left panel) and network-deformed (right panel) meshes. (Other rows) Liver surface nodes projected on x-ray projections at three projection angles. The left and right panels show the projected nodes corresponding to the prior and network-deformed surface meshes, respectively. The horizontal dash lines are added to assist visual evaluations of the liver deformation and the mesh registration.

Figure 8.

Comparison of liver Hausdorff distances of the PCA-based 2D-3D registration method and the proposed MeshRegNet-Bio method at three projection angles (0°, 45°, and 90°). For reference, the Hausdorff distance between the prior and ‘ground truth’ target liver surface meshes are presented in the first boxplot of each patient.

Figure 9.

Liver tumor overlays between the prior and the ‘ground truth’ target tumor meshes (left panel) and between the MeshRegNet-Bio deformed and target tumor meshes (right panel).

Figure 10.

Comparison of the liver tumor localization accuracy of the PCA-based and the MeshRegNet-Bio methods at three different projection angles for all 10 patient cases.