Quanti.us: a tool for rapid, flexible, crowd-based annotation of images

Abstract

We describe Quanti.us , a crowd-based image-annotation platform that provides an accurate alternative to computational algorithms for difficult image-analysis problems. We used Quanti.us for a variety of medium-throughput image-analysis tasks and achieved 10-50× savings in analysis time compared with that required for the same task by a single expert annotator. We show equivalent deep learning performance for Quanti.us-derived and expert-derived annotations, which should allow scalable integration with tailored machine learning algorithms.

Fig. 1 |. Leveraging the wisdom of crowds for scientific image analysis with Quanti.us.

a, Scientists designate a tool that human Turkers then use to annotate uploaded images according to a set of brief instructions. The resulting annotations can be interpreted in raw form and used as input to conventional algorithms, or be used as training data for machine learning algorithms. b, Left, raw example image of cell nuclei (true positives) and autofluorescent pores (true negatives). Right, corresponding overlay of expert, Turker, and clustered Turker crosshair annotations. False positive and false negative annotations were scored against those provided by a trained expert for individual Turkers, or for spatially clustered annotations from all Turkers (Methods). Each of 300 images was annotated by ten Turkers (a subset of 20 images was used to determine Turker performance). The scale bar applies to the higher-magnification (bottom) images, which represent the regions outlined by dashed squares in the corresponding images above; high magnification is 3× that in the lower-magnification image. c, Precision and recall metrics for individual Turkers (n = 46), for the clustered annotations from ten Turkers completing each image (“Turker collective”), for other experts not involved in ground truth annotation, and for a conventional FIJI object-detection pipeline over a range of particle-size thresholds. An inherent Turker quality score is shown. The gray dashed box indicates the portion of the graph highlighted in the inset to the right. Inset: the arrow indicates the effect of filtering out the bottom one-third of workers, assessed in terms of their performance, on the basis of this score. d, Annotations from every combination of a representative set of one to six ‘good’ Turkers and one ‘bad’ Turker who completed the same five image tasks were clustered and used to determine the indicated performance metrics. e, False positive errors contributed over the first k annotations submitted by a Turker (in chronological order), fit by a quadratic function (n = 29 Turkers). f, Spatial error of annotations versus the time between annotations, with Fitts’s law tradeoff (n = 129 Turkers). Fit envelopes are 95% confidence intervals. Data are representative of two experimental replicates.

Fig. 2 |. Case studies and machine learning integration of Quanti.us.

a, Left, raw example image (top) and corresponding overlay of Turker annotations and clustered annotations (bottom) of fluorescent microtubules in a gliding assay, annotated with a polyline tool. Each of 50 images was annotated by ten Turkers. Right, FIESTA output (top). Plots (bottom) show microtubule speed and length distributions. The scale bar applies to the higher-magnification images, which represent the regions outlined by dashed squares in the corresponding images; high magnification is 2.75× that in the larger, lower-magnification images. b, Top, raw image frames of a 3D z-stack spanning an organoid. Middle, raw Turker outlines of nuclei, Turker consensus outlines, expert outlines, and MINS algorithm outlines associated with one frame of the stack (outlined by a dashed rectangle). Ten Turkers annotated 30 frames. The plot in the lower right shows performance metrics (prec., precision; rec., recall) for MINS for 18 runs spanning a range of parameter settings (Methods), and for the Turker collective, relative to results from an expert. c, Left and top, raw example images and corresponding overlays of Turker annotations and clustered annotations of the nose, digits, and tail of a walking mouse (images adapted with permission from ref. , Springer Nature). Each of 29 images was annotated by 20 Turkers. We input expert or spatially clustered Turker annotations into FIJI’s TrackMate to construct gait plots (bottom) and also compared them to results of a conventional segmentation (seg.) pipeline in FIJI. “Hind” and “fore” refer to limbs. d, Left, raw example images (top) and corresponding overlays (bottom) of Turker annotations and clustered annotations for 2 of 48 frames from a movie of mammary epithelial cell spreading (ten Turkers per frame). Right, F-score plotted for five experts; the Turker collective; automated Otsu segmentation; and convolutional neural networks trained on annotations from five randomly chosen Turkers, clustered Turker annotations, or expert annotations. Data are shown as mean ± s.d. and are representative of at least two experimental replicates.