Dynamical regimes of diffusion models

Abstract

We study generative diffusion models in the regime where both the data dimension and the sample size are large, and the score function is trained optimally. Using statistical physics methods, we identify three distinct dynamical regimes during the generative diffusion process. The generative dynamics, starting from pure noise, first encounters a speciation transition, where the broad structure of the data emerges, akin to symmetry breaking in phase transitions. This is followed by a collapse phase, where the dynamics is attracted to a specific training point through a mechanism similar to condensation in a glass phase. The speciation time can be obtained from a spectral analysis of the data's correlation matrix, while the collapse time relates to an excess entropy measure, and reveals the existence of a curse of dimensionality for diffusion models. These theoretical findings are supported by analytical solutions for Gaussian mixtures and confirmed by numerical experiments on real datasets.

Fig. 1. Illustration of the three regimes of the backward dynamics through an example corresponding to a Gaussian mixture in two dimensions.

Trajectories are colored white and blue according to their class at the end of the backward dynamics. In regime I, blue and white trajectories are fluctuating within the same bundle and x is similar to white noise. At the speciation time t_S, the ensembles of blue and white trajectories divide and head towards the distribution associated to their respective class. Regime II is where the generative process constructs an x which resembles to one element of the class (e.g., a seashore in the illustration) without being linked to any data of the training set. At the collapse time t_C, trajectories start to be attracted by the data point on which they collapse at t = 0. Regime III corresponds to memorization, whereas in regime I and II, the diffusion model truly generalizes. The images on the right and on the left are illustrations obtained from our ImageNet numerical experiment (notice the collapse on the panda and seashore from the training set at t = 0).

Fig. 2. Speciation in Gaussian mixtures.

Evolution of the probability ϕ(t) that the two clones end up in the same cluster as a function of t/t_S for several values of d at fixed $\tilde{\mu }=1$ and σ = 1. The solid line corresponds to the evaluation of (6) while the dots are obtained by sampling 10,000 clone trajectories. The vertical (resp. horizontal) dashed line corresponds to t/t_S = 1 (resp. ϕ(t) = 0.775). Error bars correspond to thrice standard error.

Fig. 3. Collapse in Gaussian mixtures.

Evolution of the excess entropy density f ^e(t)/α as a function of time t for several values of d, at fixed n = 20,000. The solid lines are the theoretical predictions while the dots show the results of the numerical evaluation approximating the entropy from the dataset. The vertical dashed lines represent the collapse time t_C predicted analytically for Gaussian mixtures given in (10). Error bars correspond to thrice the standard error.

Fig. 4. Speciation in realistic datasets.

Evolution of ϕ(t), the probability that the two clones end up in the same class, as a function of t/t_S for several image datasets. The values of t_S are the theoretical prediction for the speciation time obtained using (4) and listed in Table 1. The dashed horizontal line indicate ϕ(t) = 0.775, the error bars correspond to thrice the standard error and the solid lines linearly interpolate the experimental points.

Fig. 5. Collapse in realistic datasets (ImageNet16, ImageNet32 and LSUN).

(Top-left) Evolution of ϕ_C(t), the probability that two cloned trajectories collapse on the same data of the training set at time zero. (Top-right) Histograms of ${\widehat{t}}_{\mathrm{c}}$ derived from the last-changing indices μ_⋆ on 4000 generated samples for the LSUN dataset trained with n = 200. (Bottom) Evolution of the empirical excess entropy f(t)/α. In all panels, the colored vertical dashed lines indicate the average of ${\widehat{t}}_C$. The error bars correspond to thrice the standard error.