Improving the Annotation Process in Computational Pathology: A Pilot Study with Manual and Semi-automated Approaches on Consumer and Medical Grade Devices

Abstract

The development of reliable artificial intelligence (AI) algorithms in pathology often depends on ground truth provided by annotation of whole slide images (WSI), a time-consuming and operator-dependent process. A comparative analysis of different annotation approaches is performed to streamline this process. Two pathologists annotated renal tissue using semi-automated (Segment Anything Model, SAM)) and manual devices (touchpad vs mouse). A comparison was conducted in terms of working time, reproducibility (overlap fraction), and precision (0 to 10 accuracy rated by two expert nephropathologists) among different methods and operators. The impact of different displays on mouse performance was evaluated. Annotations focused on three tissue compartments: tubules (57 annotations), glomeruli (53 annotations), and arteries (58 annotations). The semi-automatic approach was the fastest and had the least inter-observer variability, averaging 13.6 ± 0.2 min with a difference (Δ) of 2%, followed by the mouse (29.9 ± 10.2, Δ = 24%), and the touchpad (47.5 ± 19.6 min, Δ = 45%). The highest reproducibility in tubules and glomeruli was achieved with SAM (overlap values of 1 and 0.99 compared to 0.97 for the mouse and 0.94 and 0.93 for the touchpad), though SAM had lower reproducibility in arteries (overlap value of 0.89 compared to 0.94 for both the mouse and touchpad). No precision differences were observed between operators (p = 0.59). Using non-medical monitors increased annotation times by 6.1%. The future employment of semi-automated and AI-assisted approaches can significantly speed up the annotation process, improving the ground truth for AI tool development.

Keywords: Annotation; Artificial intelligence; Computational pathology; Digital pathology; Segment Anything Model.

Fig. 1

Graphic representation of the study design. Two pathologists, one with experience in annotation and the other at its beginner phase with this task, were asked to annotate three different renal structures (tubules, glomeruli, and arteries) using two manual devices (mouse vs touchpad) and a semi-automatic, AI-based tool (SAM). Starting from the masks obtained with this annotation process, performances in terms of working time required, inter-observer variability, and annotation precision were calculated, comparing the results obtained with the mouse using medical and non-medical displays

Fig. 2

The Qupath extension for SAM allows the user to draw a bounding box around the structure of interest and let the model finely annotate it

Fig. 3

The inter-annotator average times required for the annotation of different structures (tubules, violet; glomeruli, pink; arteries, light pink) using different devices (touchpad, mouse, and SAM) (A). Time variability between the two annotators is reported in %, using the same stratification (B). Finally, the reproducibility in terms of overlap fraction for ROIs obtained with different methods (C) and from the two observers (D) is reported

Fig. 4

Comparison of the annotation outlines for tubules, glomeruli, and arteries obtained with the different devices (touchpad, mouse, and SAM), with relative overlap of the ROIs obtained