NOISe: Nuclei-Aware Osteoclast Instance Segmentation for Mouse-to-Human Domain Transfer

Abstract

Osteoclast cell image analysis plays a key role in osteoporosis research, but it typically involves extensive manual image processing and hand annotations by a trained expert. In the last few years, a handful of machine learning approaches for osteoclast image analysis have been developed, but none have addressed the full instance segmentation task required to produce the same output as that of the human expert led process. Furthermore, none of the prior, fully automated algorithms have publicly available code, pretrained models, or annotated datasets, inhibiting reproduction and extension of their work. We present a new dataset with ~2 × 10⁵ expert annotated mouse osteoclast masks, together with a deep learning instance segmentation method which works for both in vitro mouse osteoclast cells on plastic tissue culture plates and human osteoclast cells on bone chips. To our knowledge, this is the first work to automate the full osteoclast instance segmentation task. Our method achieves a performance of 0.82 mAP_0.5 (mean average precision at intersection-over-union threshold of 0.5) in cross validation for mouse osteoclasts. We present a novel nuclei-aware osteoclast instance segmentation training strategy (NOISe) based on the unique biology of osteoclasts, to improve the model's generalizability and boost the mAP_0.5 from 0.60 to 0.82 on human osteoclasts. We publish our annotated mouse osteoclast image dataset, instance segmentation models, and code at github.com/michaelwwan/noise to enable reproducibility and to provide a public tool to accelerate osteoporosis research.

Figure 1.

Human osteoclast cell segmentations from our nuclei-aware osteoclast instance segmentation (NOISe) model, compared to a baseline YOLOv8 model. Since both models are trained only on mouse osteoclast data, this highlights the effectiveness of our nuclei-aware pretraining strategy for transfer learning to the human domain, where osteoclast microscope samples are harder to obtain and annotate.

Figure 2.

An illustration of an osteoclast cell in purple acting on bone structure in light brown. Osteoclasts are responsible for bone-resorption, and are characterized by having three or more nuclei and being positive for purple TRAP enzyme stain.

Figure 3.

An overview of NOISe, our nuclei-aware osteoclast instance segmentation training pipeline. A two-stage training process features a pretraining stage for multiclass detection weakly supervised by nuclei location information. The pretraining significantly boosts subsequent performance of the overall osteoclast instance segmentation model, especially in the data-scarce human domain.

Figure 4.

Detail from a mouse osteoclast microscope image, together with weak ground truth nuclei labels (note the uniform size and shape of the boxes), and YOLOv8 object detection predictions of the same. Our nuclei-aware training method exploits this weak ground truth information to improve the generalizability of osteoclast instance segmentation.

Figure 5.

Cell images and cropped patches from mouse experiments M1 through M5, and human experiments H1 and H2, illustrating the diversity in slide lighting, background, and appearance, and in osteoclast size, shape, and density in our data.

Figure 6.

Predictions of our mouse osteoclast instance segmentation models under five-fold cross-validation. Ground truth osteoclast shape masks are carefully produced by experts. Our model’s predictions of these masks are generally faithful, but we highlight some mistakes and challenges. In (b), image blur seems to impair the model’s overall ability; in (c), a pre-osteoclast on the top left is mistakenly identified as an osteoclast; and in (d), an osteoclast on the top left is not detected, while two pre-osteoclasts below are mis-identified as osteoclasts.

Figure 7.

Qualitative results from different training configurations of YOLOv8 and NOISe models tested on human osteoclast data, showing that the NOISe strategy can improve performance on human osteoclasts even when the model is trained only on mouse data (NOISe M→H).