Objective comparison of particle tracking methods

Abstract

Particle tracking is of key importance for quantitative analysis of intracellular dynamic processes from time-lapse microscopy image data. Because manually detecting and following large numbers of individual particles is not feasible, automated computational methods have been developed for these tasks by many groups. Aiming to perform an objective comparison of methods, we gathered the community and organized an open competition in which participating teams applied their own methods independently to a commonly defined data set including diverse scenarios. Performance was assessed using commonly defined measures. Although no single method performed best across all scenarios, the results revealed clear differences between the various approaches, leading to notable practical conclusions for users and developers.

Figure 1. Simulated image data.

Representative images of the three main factors (particle dynamics, density and signal) affecting tracking performance are shown. (a) Four biological scenarios were simulated, of which we show snapshot images (i–iv) and trajectories (v–viii) in arbitrary colors: particles showing random-walk motion imaged in two dimensions over time (2D+time) using wide-field microscopy (i,v); larger (elongated) particles represented by asymmetric Gaussians showing directed motion in 2D+time (ii,vi); particles switching between random-walk and randomly oriented directed motion imaged in 2D+time using confocal microscopy (iii,vii); and particles switching between random-walk and directed motion with restricted orientation imaged in 3D+time (only one slice is shown) using confocal microscopy (iv,viii). (b,c) Three density levels (b; low, medium and high) and four SNR levels (c; 1, 2, 4 and 7) were simulated.

Figure 2. Sample performance results.

Values of three performance measures (α, β and RMSE) are plotted as a function of density (low, medium and high) and SNR for scenario 1. (a) α values (scoring the match between ground-truth and estimated tracks) for each density. (b) β values (α values with a penalty for nonmatching estimated tracks) for each density. (c) RMSE values (scoring localization accuracy) for each density. For some methods, the lines are incomplete, indicating missing (not submitted) tracking results. Source data

Figure 3. The top three best-performing methods for each performance measure and combination of biological scenario, particle density and SNR.

The cells are color coded according to method number (Table 1).