Real-time CBCT imaging and motion tracking via a single arbitrarily-angled x-ray projection by a joint dynamic reconstruction and motion estimation (DREME) framework

Abstract

Objective.Real-time cone-beam computed tomography (CBCT) provides instantaneous visualization of patient anatomy for image guidance, motion tracking, and online treatment adaptation in radiotherapy. While many real-time imaging and motion tracking methods leveraged patient-specific prior information to alleviate under-sampling challenges and meet the temporal constraint (<500 ms), the prior information can be outdated and introduce biases, thus compromising the imaging and motion tracking accuracy. To address this challenge, we developed a frameworkdynamicreconstruction andmotionestimation (DREME) for real-time CBCT imaging and motion estimation, without relying on patient-specific prior knowledge.Approach.DREME incorporates a deep learning-based real-time CBCT imaging and motion estimation method into a dynamic CBCT reconstruction framework. The reconstruction framework reconstructs a dynamic sequence of CBCTs in a data-driven manner from a standard pre-treatment scan, without requiring patient-specific prior knowledge. Meanwhile, a convolutional neural network-based motion encoder is jointly trained during the reconstruction to learn motion-related features relevant for real-time motion estimation, based on a single arbitrarily-angled x-ray projection. DREME was tested on digital phantom simulations and real patient studies.Main Results.DREME accurately solved 3D respiration-induced anatomical motion in real time (∼1.5 ms inference time for each x-ray projection). For the digital phantom studies, it achieved an average lung tumor center-of-mass localization error of 1.2 ± 0.9 mm (Mean ± SD). For the patient studies, it achieved a real-time tumor localization accuracy of 1.6 ± 1.6 mm in the projection domain.Significance.DREME achieves CBCT and volumetric motion estimation in real time from a single x-ray projection at arbitrary angles, paving the way for future clinical applications in intra-fractional motion management. In addition, it can be used for dose tracking and treatment assessment, when combined with real-time dose calculation.

Keywords: deep learning; dynamic CBCT reconstruction; motion estimation; motion model; real-time imaging; x-ray.

Figure 1.

Workflow for dynamic CBCT reconstruction and real-time motion estimation. DREME uses cone-beam projections from a standard pre-treatment CBCT scan as input to simultaneously reconstruct a sequence of dynamic CBCTs (upper route), and a patient reference anatomy and its corresponding motion model (lower route) which are subsequently used for the real-time CBCT imaging and target tracking during treatment.

Figure 2.

DREME network architecture. (a) DREME consists of a learnable hash encoder, a spatial implicit neural representation (INR), a learnable B-spline interpolant, and a convolutional neural network (CNN)-based motion encoder. It adopts a motion-compensated CBCT reconstruction approach based on a pre-treatment cone-beam acquisition $\mathit{p}$, by joint deformable registration and reconstruction. The hash encoder and INR estimate the reference CBCT ${\mathit{I}}_{\text{ref}}\left(\mathit{x}\right)$, while the CNN-based motion encoder and the learnable B-spline interpolant estimate the projection-dependent (temporal) and spatial components of the deformation vector fields $\mathit{d}\left(\mathit{x},p\right)$ (DVFs) with respect to the reference CBCT. Dynamic CBCTs I(x, p) are derived by deforming the reference CBCT using the solved DVFs corresponding to each projection p of the cone-beam acquisition $\mathit{p}$. (b) The spatial INR uses a three-layer multilayer perceptron (MLP) with periodic activation functions to represent the mapping from the voxel coordinate x to the attenuation coefficient ${\mathit{I}}_{\text{ref}}\left(\mathit{x}\right)$. N_ft denotes the feature number in each layer. (c) The CNN-based motion encoder takes an x-ray projection p as an input, extracting motion-related image features to estimate the corresponding MBC scores (i.e., temporal coefficients) ${\mathit{w}}_i\left(p\right)$.

Figure 3.

Learning task 1: dynamic CBCT reconstruction. The learning task of dynamic CBCT reconstruction is driven by maximizing the similarity between the digitally reconstructed radiographs (DRRs) of the motion-resolved CBCTs and the corresponding cone-beam projections. To regularize the ill-posed spatiotemporal inverse problem, regularization losses are incorporated into the training objectives.

Figure 4.

Learning task 2: real-time motion estimation. For the learning task of real-time imaging, deformable motion augmentation simulates random motion states by randomizing and resampling the MBC scores ${\mathit{w}}_i\left(p\right)$ to enhance the model’s robustness to unseen motion states. In addition, projection angle augmentation is implemented to simulate DRRs of motion-augmented CBCTs at random projection angles, to promote angle-agnostic learning.

Figure 5.

Onboard inference stage for real-time CBCT imaging and target localization. During the onboard inference stage, the CNN-based motion encoder takes an onboard projection p at an arbitrary angle as input, and estimates the MBC scores ${\mathit{w}}_i\left(p\right)$ to derive real-time CBCT $\mathit{I}\left(\mathit{x},p\right)$ or 3D target.

Figure 6.

Lung tumor motion trajectories along the superior-inferior (SI) direction in the XCAT simulation study. Trajectories X1–X6 shared the same maximum motion ranges in the SI direction, whereas the SI range of X7 was extended to evaluate the robustness of DREME models trained using the other scenarios (X1–X6).

Figure 7.

Comparison of reconstructed reference CBCTs in the comparison study. 2D3D_PCA-4DCT assumed the existence of an artifact-free pre-treatment 4D-CT scan and used the end-of-exhale phase as the reference anatomy. 2D3D_PCA-4DCBCT and DREME reconstructed the reference CBCTs using the pre-treatment scans, thus the anatomy is up-to-date and specific to daily motion scenarios. The reference CBCTs of 2D3D_PCA-4DCT were reconstructed from the end-of-exhale phase after projection phase sorting, suffering from significant under-sampling and motion-related artifacts. DREME reconstructed the reference CBCTs with simultaneous motion estimation/compensation, resulting in minimal motion-related artifacts.

Figure 8.

Tumor motion trajectories of the comparison study. The first and second columns respectively presents the comparison of the solved tumor motion for the X1–X7 scenarios in the superior-inferior (SI) and anterior-posterior (AP) directions between 2D3D_PCA-4DCBCT, 2D3D_PCA-4DCT, DREME, and the ‘ground-truth’ reference. DREME was trained on the X3 scenario.

Figure 9.

Examples of real-time CBCTs and lung tumor motion solved by DREME for the XCAT study: (a) X5 scenario and (b) X6 scenario. The training scenario was X3. The first row shows the tumor motion curves along the SI direction, with the dots indicating the time points selected for plotting. The second row presents the onboard x-ray projections at the selected time points, and the ‘×’ symbols indicate the solved tumor center-of-mass positions projected onto the projections. In the following rows, real-time CBCTs of the selected time points are compared against the ‘ground-truth’ CBCTs, with the difference images calculated. The estimated tumor contours (red) are also presented for each selected time points.

Figure 10.

Reference CBCTs reconstructed by DREME for the patient study. Left and right columns present the axial and coronal views of the reference CBCTs for the full-fan and half-fan scans, respectively. Except for P8, whose CBCT scan covered the abdominal region, the other CBCT scans covered the thoracic region.

Figure 11.

Comparison of the reference (red solid line) and DREME estimated (black dashed line) motion trajectories extracted using the AS method for the patient study. For some patients, the tracking targets may move out of the field of view, thus only partial trajectories were extracted from their AS images.

Figure 12.

Real-time CBCTs and motion trajectories estimated by DREME for (a) P1 and (b) P2 of the patient study. The first row shows the SI motion trajectory of the tracking landmark (P1: lung nodule; P2: diaphragm apex), with the dots indicating the acquisition time points selected for plotting. The second row presents the onboard x-ray projections at the selected time points, and the ‘□’ and ‘+’ symbols indicate the 2D-projected landmark positions solved by DREME. In the following rows, real-time CBCTs at the selected time points are presented in the coronal and sagittal views.