Real-time CBCT Imaging and Motion Tracking via a Single Arbitrarily-angled X-ray Projection by a Joint Dynamic Reconstruction and Motion Estimation (DREME) Framework

Abstract

Objective: Real-time cone-beam computed tomography (CBCT) provides instantaneous visualization of patient anatomy for image guidance, motion tracking, and online treatment adaptation in radiotherapy. While many real-time imaging and motion tracking methods leveraged patient-specific prior information to alleviate under-sampling challenges and meet the temporal constraint (< 500 ms), the prior information can be outdated and introduce biases, thus compromising the imaging and motion tracking accuracy. To address this challenge, we developed a framework (DREME) for real-time CBCT imaging and motion estimation, without relying on patient-specific prior knowledge.

Approach: DREME incorporates a deep learning-based real-time CBCT imaging and motion estimation method into a dynamic CBCT reconstruction framework. The reconstruction framework reconstructs a dynamic sequence of CBCTs in a data-driven manner from a standard pre-treatment scan, without utilizing patient-specific knowledge. Meanwhile, a convolutional neural network-based motion encoder is jointly trained during the reconstruction to learn motion-related features relevant for real-time motion estimation, based on a single arbitrarily-angled x-ray projection. DREME was tested on digital phantom simulation and real patient studies.

Main results: DREME accurately solved 3D respiration-induced anatomic motion in real time (~1.5 ms inference time for each x-ray projection). In the digital phantom study, it achieved an average lung tumor center-of-mass localization error of 1.2±0.9 mm (Mean±SD). In the patient study, it achieved a real-time tumor localization accuracy of 1.8±1.6 mm in the projection domain.

Significance: DREME achieves CBCT and volumetric motion estimation in real time from a single x-ray projection at arbitrary angles, paving the way for future clinical applications in intra-fractional motion management. In addition, it can be used for dose tracking and treatment assessment, when combined with real-time dose calculation.

Keywords: Deep learning; Dynamic CBCT reconstruction; Motion estimation; Motion model; Real-time imaging; X-ray.

Figure 1.

The DREME network architecture for dynamic CBCT reconstruction and real-time motion estimation. (a) DREME adopts a motion-compensated CBCT reconstruction approach based on a standard pre-treatment cone-beam acquisition $\mathit{p}$, by joint deformable registration and reconstruction. The spatial implicit neural representation (INR) estimates the reference-frame CBCT ${\mathit{I}}_{\mathit{\text{ref}}}\left(\mathit{x}\right)$, and the CNN-based motion encoder, together with the learnable B-spline interpolant, estimates the deformation vector fields (DVFs) with respect to the reference CBCT. The dynamic CBCTs $\mathit{I}(\mathit{x},\mathit{p})$ are derived by deforming the reference CBCT using the solved DVFs corresponding to each projection $p$ of the cone-beam acquisition $\mathit{p}$. The learning task of dynamic CBCT reconstruction is driven by maximizing the similarity between the digitally reconstructed radiographs (DRRs) and the corresponding cone-beam projections. To regularize the ill-posed spatiotemporal inverse problem, regularization losses are incorporated into the training objectives. (b) For the learning task of real-time imaging, deformable motion augmentation simulates random motion states by resampling MBC scores ${\mathit{w}}_i\left(p\right)$ to enhance the model’s robustness to unseen motion states. In addition, projection angle augmentation is implemented to simulate DRRs of motion-augmented CBCTs at random projection angles, to promote angle-agnostic learning. (c) During the onboard inference stage, the motion encoder takes an onboard projection at an arbitrary angle as input and outputs the MBC scores to derive the real-time CBCT or 3D target geometry, based on the patient anatomy ${\mathit{I}}_{\mathit{\text{ref}}}\left(\mathit{x}\right)/\text{tracking}$ target and the motion model (MBCs).

Figure 1.

The DREME network architecture for dynamic CBCT reconstruction and real-time motion estimation. (a) DREME adopts a motion-compensated CBCT reconstruction approach based on a standard pre-treatment cone-beam acquisition $\mathit{p}$, by joint deformable registration and reconstruction. The spatial implicit neural representation (INR) estimates the reference-frame CBCT ${\mathit{I}}_{\mathit{\text{ref}}}\left(\mathit{x}\right)$, and the CNN-based motion encoder, together with the learnable B-spline interpolant, estimates the deformation vector fields (DVFs) with respect to the reference CBCT. The dynamic CBCTs $\mathit{I}(\mathit{x},\mathit{p})$ are derived by deforming the reference CBCT using the solved DVFs corresponding to each projection $p$ of the cone-beam acquisition $\mathit{p}$. The learning task of dynamic CBCT reconstruction is driven by maximizing the similarity between the digitally reconstructed radiographs (DRRs) and the corresponding cone-beam projections. To regularize the ill-posed spatiotemporal inverse problem, regularization losses are incorporated into the training objectives. (b) For the learning task of real-time imaging, deformable motion augmentation simulates random motion states by resampling MBC scores ${\mathit{w}}_i\left(p\right)$ to enhance the model’s robustness to unseen motion states. In addition, projection angle augmentation is implemented to simulate DRRs of motion-augmented CBCTs at random projection angles, to promote angle-agnostic learning. (c) During the onboard inference stage, the motion encoder takes an onboard projection at an arbitrary angle as input and outputs the MBC scores to derive the real-time CBCT or 3D target geometry, based on the patient anatomy ${\mathit{I}}_{\mathit{\text{ref}}}\left(\mathit{x}\right)/\text{tracking}$ target and the motion model (MBCs).

Figure 2.

Lung tumor motion trajectories along the superior-inferior (SI) direction in the XCAT simulation study. Trajectories X1-X6 shared the same maximum motion ranges in the SI direction, whereas the SI range of X7 were extended to evaluate the robustness of DREME models trained using the other scenarios (X1-X6).

Figure 3.

Comparison of reconstructed reference CBCTs in the comparison study. 2D3D_PCA-4DCT assumed the existence of an artifact-free pre-treatment 4D-CT scan and used the end-of-exhale phase as the reference anatomy. 2D3D_PCA-4DCBCT and DREME reconstructed the reference CBCTs using the pre-treatment scans, thus the anatomy is up-to-date and specific to daily motion scenarios. The reference CBCTs of 2D3D_PCA-4DCT were reconstructed from the end-of-exhale phase after projection phase sorting, suffering from significant under-sampling and motion-related artifacts. DREME reconstructed the reference CBCTs with simultaneous motion estimation/compensation, resulting in minimal motion-related artifacts.

Figure 4.

Tumor motion trajectories of the comparison study. The first and second columns respectively presents the comparison of the solved tumor motion for the X1-X7 scenarios in the SI and AP directions between 2D3D_PCA-4DCBCT, 2D3D_PCA-4DCT, DREME, and the ‘ground-truth’ reference. DREME was trained on the X3 scenario.

Figure 5.

Examples of real-time CBCTs and lung tumor motion solved by DREME for the XCAT study: (a) X5 scenario and (b) X6 scenario. The training scenario was X3. The first row shows the tumor motion curves along the SI direction, with the dots indicating the time points selected for plotting. The second raw presents the onboard x-ray projections at the selected time points, and the “x” symbols indicate the solved tumor center-of-mass positions projected onto the projections. In the following rows, real-time CBCTs of the selected time points are compared against the ‘ground-truth’ CBCTs, with the difference images calculated. The estimated tumor contours (red) are also presented for each selected time points.

Figure 6.

Real-time CBCTs and motion trajectories estimated by DREME for (a) P1 and (b) P2 of the patient study. The first row shows the SI motion trajectory of the tracking landmark (P1: lung nodule; P2: diaphragm apex), with the dots indicating the acquisition time points selected for plotting. The second raw presents the onboard x-ray projections at the selected time points, and the “□” and “+” symbols indicate the projected landmark positions in the projections. In the following rows, real-time CBCTs at the selected time points are presented in the coronal and sagittal views.