Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software

Abstract

With the widespread uptake of two-dimensional (2D) and three-dimensional (3D) single-molecule localization microscopy (SMLM), a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort, we designed a competition to extensively characterize and rank the performance of 2D and 3D SMLM software packages. We generated realistic simulated datasets for popular imaging modalities-2D, astigmatic 3D, biplane 3D and double-helix 3D-and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D SMLM software and provides a holistic view of how the latest 2D and 3D SMLM packages perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against the current state of the art, and provides insight into the current limits of the field.

Figure 1:. Summary of SMLM challenge simulations.

A. 3D rendering of simulated microtubules and endoplasmic reticulum samples. B. Key simulation steps. The structure is constructed from 3D tubes continuously defined by three B-spline functions in the volume of interest. Membranes of the tubes are densely populated with possible positions. Fluorophores follow a 4-state photophysics model. Activations of a given frame are convolved with the experimental PSF and shot & camera noise is added. C. Summary of all 16 challenge datasets, calibration data and experimental PSFs. Left column: orthogonal projections of the experimentally-derived PSF. Right column: exemplar frame for each competition dataset, characterized by structure (endoplasmic reticulum, E; microtubules, MT), modality (2D; astigmatism, AS; double helix, DH; biplane, BP), density (low density, LD; high density, HD) and SNR (noise level N1, N2, N3). BP Ch. 1,2, indicates two biplane channels with a relative focal shift of 500 nm.

Figure 2:. Leaderboards for each competition modality, at low and high spot density.

Ranking is based on software Efficiency, which combines Jaccard index (fraction of successfully detected molecules) and localization precision (RMSE, root mean square error, lateral & axial). Orange, contribution of high SNR dataset; blue, contribution of low SNR dataset.

Figure 3:. Comparison of 3D software performance.

Gold stars indicate top performers for each dataset. Dashed lines in top, middle panels indicate overall efficiency (higher is better). A-C. Localization error and spot detection performance of all astigmatic SMLM software. D-E. Average (colored marker with s.d. error bars, sample sizes for each category indicated in Supplementary Table 2) and best-in-class (colored marker with gold star) software performance for all competition modalities. AS, astigmatism; DH, double helix; BP, biplane.

Figure 4:. Super-resolved images of software results for simulated and real competition datasets.

A. Xy and xz projection images of 3D competition datasets for representative software. Top: best-in-class software in each modality, for high SNR low density dataset. Bottom: representative average software. Left: xy and xz overview images for winning AS software. Middle: xy and xz zoom images of boxed regions in left panel, for winning and mid-range software, each modality. Right: xy and xz line profiles of winning and mid-range software for each modality, for boxed regions in middle panel. Image colors: red, ground truth; green, software results. Line profiles: GT, ground truth, black; AS, astigmatism, red; BP, biplane, blue; DH, double helix, green. Panel key: Software-name Dataset-ranking°. Scale bar: full image, 1 μm, magnified regions, 100 nm. B. Astigmatism software results for real nuclear pore complex 3D STORM data. Top: Super-resolved overview image in xy for 3D-DAOSTORM software, color coded for depth. Bottom: xz orthoslices along 600 nm wide dashed region indicated in top panel for 8 astigmatism software packages. Scale bars, 500 nm.