A data-driven approach for real-time soft tissue deformation prediction using nonlinear presurgical simulations

Abstract

A method that allows a fast and accurate registration of digital tissue models obtained during preoperative, diagnostic imaging with those captured intraoperatively using lower-fidelity ultrasound imaging techniques is presented. Minimally invasive surgeries are often planned using preoperative, high-fidelity medical imaging techniques such as MRI and CT imaging. While these techniques allow clinicians to obtain detailed 3D models of the surgical region of interest (ROI), various factors such as physical changes to the tissue, changes in the body's configuration, or apparatus used during the surgery may cause large, non-linear deformations of the ROI. Such deformations of the tissue can result in a severe mismatch between the preoperatively obtained 3D model and the real-time image data acquired during surgery, potentially compromising surgical success. To overcome this challenge, this work presents a new approach for predicting intraoperative soft tissue deformations. The approach works by simply tracking the displacements of a handful of fiducial markers or analogous biological features embedded in the tissue, and produces a 3D deformed version of the high-fidelity ROI model that registers accurately with the intraoperative data. In an offline setting, we use the finite element method to generate deformation fields given various boundary conditions that mimic the realistic environment of soft tissues during a surgery. To reduce the dimensionality of the 3D deformation field involving thousands of degrees of freedom, we use an autoencoder neural network to encode each computed deformation field into a short latent space representation, such that a neural network can accurately map the fiducial marker displacements to the latent space. Our computational tests on a head and neck tumor, a kidney, and an aorta model show prediction errors as small as 0.5 mm. Considering that the typical resolution of interventional ultrasound is around 1 mm and each prediction takes less than 0.5 s, the proposed approach has the potential to be clinically relevant for an accurate tracking of soft tissue deformations during image-guided surgeries.

Fig 1. Flowchart detailing training and deployment of the entire pipeline.

(a) Starting with a geometric model of the patient’s region of interest (ROI), we create a deformation dataset with FEM from a widely varying force field. This dataset contains the displacements of all nodes in the model, but can also be used to track just the displacement of the fiducial markers (FMs). (b) Using the full deformation dataset, we train the autoencoder to create a latent space representation which encodes the shape deformations. (c) Also during the training phase, a neural network (NN) and ridge regression (RR) model are trained to map FM displacements to the latent space. (d) In the deployment phase, the NN or RR is used to map in situ FM displacements to the latent space representation. The trained decoder reconstructs the full shape deformation from the latent vector.

Fig 2. CT reconstructed geometric models used in this study.

The models pictured are: (a) head and neck tumor, (b) head and neck tumor cross-section, (c) kidney, (d) kidney cross-section, (e) aorta aneurysm, and (e) aorta aneurysm cross-section.

Fig 3. Visualization of fLSC and intermediate force fields on the H&N tumor during the force Laplace smoothing process.

The procedures of generating $f_{LSC}$ are: (i) generate concentrated forces with a randomly sampled magnitude between  [ − s , s ]  and a randomly assigned node on the outer surface as the loading position, and (ii) use LBO to locally smooth the concentrated forces and generate a smoothed concentrated force field ($f_{LSC}$). In the figure, the initial magnitude of three forces: $[f_x,f_y,f_z]=[5.0,5.0,5.0]$ N, and $f_{LSC}$ is eventually obtained after 20 iterations of smoothing with a rate of 0.1.

Fig 4. Minimum, median, and maximum deformations for all three geometries along with undeformed geometry and NN reconstructed deformation.

The gray mesh corresponds to the undeformed geometry, the blue mesh corresponds to the ground truth deformation, and the yellow mesh corresponds to the NN reconstruction.

Fig 5. Violin plots comparing reconstruction error between the NN and RR models on the test set for all three geometries.

Red corresponds to the NN model, green corresponds to the RR model, and quartiles are marked by the dashed lines. (a) Mean nodal reconstruction error. (b) Maximum nodal reconstruction error.

Fig 6. Reconstructed deformations on all three geometries for NN and RR models, corresponding to the minimum, median, and maximum error cases between the reconstructed and ground truth deformations.

The blue mesh corresponds to the ground truth deformation, the green mesh corresponds to the NN reconstruction, and the red mesh corresponds to the RR reconstruction.

Fig 7. Parametric study results for varying numbers of FMs and latent dimensions on the H&N tumor model.

The average nodal displacement errors (i.e., mean nodal error and max nodal error) are plotted together with error bars. (a) Varying number of FMs, with latent dimension of 5. (b) Varying latent dimension with 5 FMs.