Calibration-Jitter: Augmentation of hyperspectral data for improved surgical scene segmentation

Abstract

Semantic surgical scene segmentation is crucial for accurately identifying and delineating different tissue types during surgery, enhancing outcomes and reducing complications. Hyperspectral imaging provides detailed information beyond visible color filters, offering an enhanced view of tissue characteristics. Combined with machine learning, it supports critical tumor resection decisions. Traditional augmentations fail to effectively train machine learning models on illumination and sensor sensitivity variations. Learning to handle these variations is crucial to enable models to better generalize, ultimately enhancing their reliability in deployment. In this article, Calibration-Jitter is introduced, a spectral augmentation technique that leverages hyperspectral calibration variations to improve predictive performance. Evaluated on scene segmentation on a neurosurgical dataset, Calibration-Jitter achieved a F1-score of 74.35% with SegFormer, surpassing the previous best of 70.2%. This advancement addresses limitations of traditional augmentations, improving hyperspectral imaging segmentation performance.

Keywords: biomedical imaging; brain; image segmentation.

FIGURE 1

Hyperspectral data acquisition setup for surgical applications. Left: Standard data acquisition setup, consisting of a hyperspectral camera mounted on an articulated arm or robotic system, positioned above the surgical field to capture hyperspectral images of the target tissue or organ. Illumination sources can be broadband light sources or specialized hyperspectral systems. Right: Calibration setup for acquiring dark reference with the illumination source off, white reference using a standardized white reference target, and raw sample measurement. The reference measurements are used to calibrate the acquired hyperspectral data for comparable spectral analysis and tissue characterization.

FIGURE 2

Example pipeline of dataloader. Hyperspectral data is first calibrated and then augmented. With the proposed method, we augment the white reference in step 1.

FIGURE 3

Examples of augmentation on the same image. (a) RGB image with labelled points. (b) ground truth annotation. (c) Psuedo RGB and spectra of corresponding points at labelled points for coefficient of 0.1. (d) Psuedo RGB and spectra of corresponding points at labelled points for coefficient of 0, that is, normal white‐dark calibration. (e) Psuedo RGB and spectra of corresponding points at labelled points for coefficient of −0.1. Psuedo RGB is formed by the closest available wavelengths to 630 nm (red), 540 nm (green), 480 nm (blue). (f)–(h) Higher, lower and zero coefficient (0.1, −0.1 and 0, respectively) for the corresponding class. Alongside the classes mean and standard deviation over the whole dataset. (f) Normal class, (g) tumour class and (h) blood vessel class.

FIGURE 4

Qualitative results from SegFormer model with and without augmentation applied. First column is the ground truth annotation. Second column is the RGB image corresponding to the ground truth. Third column is normal augmentation (no Calibration‐Jitter) and last column is with Calibration‐Jitter applied. The Calibration‐Jitter images have significantly less False Positives tumor predictions and even under difficult lighting conditions the Normal and Blood vessels could still be identified as seen in row 3 Jitter.