Ultrasound-guided needle tracking with deep learning: A novel approach with photoacoustic ground truth

Abstract

Accurate needle guidance is crucial for safe and effective clinical diagnosis and treatment procedures. Conventional ultrasound (US)-guided needle insertion often encounters challenges in consistency and precisely visualizing the needle, necessitating the development of reliable methods to track the needle. As a powerful tool in image processing, deep learning has shown promise for enhancing needle visibility in US images, although its dependence on manual annotation or simulated data as ground truth can lead to potential bias or difficulties in generalizing to real US images. Photoacoustic (PA) imaging has demonstrated its capability for high-contrast needle visualization. In this study, we explore the potential of PA imaging as a reliable ground truth for deep learning network training without the need for expert annotation. Our network (UIU-Net), trained on ex vivo tissue image datasets, has shown remarkable precision in localizing needles within US images. The evaluation of needle segmentation performance extends across previously unseen ex vivo data and in vivo human data (collected from an open-source data repository). Specifically, for human data, the Modified Hausdorff Distance (MHD) value stands at approximately 3.73, and the targeting error value is around 2.03, indicating the strong similarity and small needle orientation deviation between the predicted needle and actual needle location. A key advantage of our method is its applicability beyond US images captured from specific imaging systems, extending to images from other US imaging systems.

Keywords: Deep learning; Needle tracking; Photoacoustic imaging; Ultrasound imaging.

Fig. 1

Network architecture for improving needle visibility in US imaging. (a) The backbone architecture of UIU-Net, (b) the architecture of small U-Net which is incorporated into the backbone, represented by lilac arrows in (a), (c) the architecture of small U-Net which is incorporated into the backbone in the last two scales, represented by light red arrows in (a), (d) detailed structure of IC-A module yielding cross channel feature F_C-A, represented by skin-color boxes in (a), (e) detailed structure of IC-A module yielding interactive-cross spatial attention feature F_IC-A, represented by light blue boxes in (a).

Fig. 2

Imaging system descriptions and representative US/PA images. (a) Imaging systems used to capture both US and PA images. Inset: photograph of the handheld probe combining optical fiber, US transducer, and the holder, OF optical fiber. (b) Representative US images during needle insertion, where yellow arrows mark the needle position. (c) Representative PA images during needle insertion. (d) Preprocessed PA images which served as ground truth for network training. (b-d) Have same scale bar as shown in one of the figures.

Fig. 3

Representative images from the test dataset with needle insertions into chicken tissue. (a) conventional US images, (b) overlaid US image with preprocessed PA (served as ground truth), (c) overlaid US image with the prediction from traditional U-Net, (d) overlaid US image with the prediction from Attention U-Net, (e) overlaid US image with the prediction from R2U-Net, (f) overlaid US image with the prediction from UIU-Net. All figures have same scale bar, as shown in (a).

Fig. 4

Representative images from a separate experiment (unseen data) with 18 G needle insertions into chicken tissue. (a) conventional US images, (b) overlaid US image with preprocessed PA (served as ground truth for comparison), (c) overlaid US image with the prediction from traditional U-Net, (d) overlaid US image with the prediction from Attention U-Net, (e) overlaid US image with the prediction from R2U-Net, (f) overlaid US image with the prediction from UIU-Net. All figures have same scale bar, as shown in (a).

Fig. 5

Representative images from a separate experiment (unseen data) with 23 G needle insertions into chicken tissue. (a) conventional US images, (b) overlaid US image with preprocessed PA (served as ground truth for comparison), (c) overlaid US image with the prediction from traditional U-Net, (d) overlaid US image with the prediction from Attention U-Net, (e) overlaid US image with the prediction from R2U-Net, (f) overlaid US image with the prediction from UIU-Net. All figures have same scale bar, as shown in (a).

Fig. 6

Representative images with needle insertions into the human body. (a) conventional US images, (b) overlaid US image with ground truth (manual label shown in green), (c) overlaid US image with prediction from UIU-Net.