Image-guided intraoperative cortical deformation recovery using game theory: application to neocortical epilepsy surgery

Abstract

During neurosurgery, nonrigid brain deformation prevents preoperatively-acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track intraoperative cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration from the surface deformation estimation, a framework that can solve for disparate and often competing variables is needed. Game theory, which was developed to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, game theory is applied to cortical surface tracking during neocortical epilepsy surgery and used to infer information about the physical processes of brain surface deformation and image acquisition. The method is successfully applied to eight in vivo cases, resulting in an 81% decrease in mean surface displacement error. This includes a case in which some of the initial camera calibration parameters had errors of 70%. Additionally, the advantages of using a game theoretic approach in neocortical epilepsy surgery are clearly demonstrated in its robustness to initial conditions.

Fig. 1

Intraoperative image indicating the misalignment of the projected sulci (green) with the intraoperative imaged sulci positions (black), due to either camera calibration errors (left) or cortical surface deformation (right).

Fig. 2

The game theoretic algorithm uses intraoperative information to guide the estimation of cortical surface deformation and update camera calibration parameters. This algorithm was developed using a Bayesian approach.

Fig. 3

Left: A view of the surgical camera embedded in the OR light fixture during a phantom experiment. The overall setup is the same for surgical applications. Right: The method of acquiring fiducial points in the OR and an example of typically-acquired images.

Fig. 4

Mean and maximum displacement (blue) with mean and maximum residual error (red) for eight intraoperative neocortical epilepsy data sets. The algorithm was implemented with α = 4, β = 0.83, ν = 0.1, and η = 25 for all cases. (See Appendices A and B for algorithm equations.)

Fig. 5

A: Preoperative Surface of Data Sets 4 and 5, Acquired from MRI. The region that deforms during surgery is highlighted in yellow and features used to aid the algorithm are shown in black. B: Cortical Surface Deformation Results at Two Time Points. The purple and blue surfaces represent the algorithm tracking results for data sets 4 (2 hours into surgery) and 5 (3.25 hours into surgery), respectively. The distance between the two surfaces shows the cortical shift in that time, as calculated by the game theoretic algorithm. Sulci used as features are again highlighted in black. The surfaces mainly deformed in the direction of gravity, as indicated by the arrow.

Fig. 6

Data Set #4, Two Hours into Surgery. Left: Intraoperative Stereo Images of the Cortical Surface. The projected predicted features (yellow) are better aligned with the features outlined from the intraoperative images (black) than the projected preoperative features (green). Right: Intraoperative Cortical Surface with Backprojected Intensities. Backprojection is achieved by using the calibration parameters to project each element of the surface to one of the stereo images and assigning the corresponding image intensities to the surface elements. (See Appendix A.) The stereo images were manually cropped to include only brain surface intensities. Therefore, each of the surface elements was either assigned one of these image intensities or appears black, meaning no intensity was assigned. The predicted cortical surface reduces the mismatch, shown by the red arrows, between the backprojected sulci and the sulci lying on the surface (green/yellow).

Fig. 7

Data Set #5, 3.25 Hours into Surgery Left: Intraoperative Stereo Images of the Cortical Surface. The initial calibration of the camera 1 (right) is so inaccurate, that the features (green) do not project to the image field of view. The algorithm is able to correct this error and predict the cortical surface to within 1 mm. Right: Intraoperative Cortical Surface with Backprojected Intensities. Most of the right stereo image backprojected intensities do not fall on the preoperative surface due to calibration inaccuracies and the surface position. With the updated calibration and deformation parameters, intensities and features from both images are better aligned, though there is still some distortion outside the calibration region.

Fig. 8

Data Set #8 Left: Intraoperative Stereo Images of the Cortical Surface. Preoperative sulci projected with initial calibration parameters are shown in green. The algorithm is able to correct some mismatch between these and the imaged sulcal locations (black). The residual misalignment of predicted projected features (yellow) is most likely due to the poor positioning of the electrode grid. Right: Intraoperative Cortical Surface with Backprojected Intensities. Overall, the game theoretic result for data set #8 decreased the tracking error and improved the deformation estimation. Evidence of this can be seen in the reduction of feature-matching error shown by the red arrows. However, some feature mismatch (blue arrows) is still evident in the backprojected images from the algorithm-predicted surfaces.

Fig. 9

Electrode Grid Used for Calibration of Data Set #8. The shape of the brain and the size of the dural opening prevented the grid from lying flat on the brain surface.

Fig. 10

Algorithm Robustness Comparison. Left: Sensitivity to Initial Surface Position. Right: Sensitivity to Initial Camera Calibration Errors.

Fig. 11

Pictorial Description of the Feature Term. The far left image shows the brain surface as extracted from the preoperative MRI and the region of that surface (surface patch) that deforms during surgery. Using a projection function, 3D sulci from the surface patch are projected onto the 2D stereo camera images. Assuming an accurate camera calibration, the mismatch between the projected features (cyan) and the imaged features (red) is due to surface deformation. The right orange boxes show the effects of various displacement fields applied to the surface patch. The distance between the projected sulci from those patches (yellow) and the imaged sulci (red) is measured for each applied displacement. In this case, the displacement field applied to the patch on the far right is the most likely because it best aligns the two sets of features.