Archetypal landscapes for deep neural networks

Abstract

The predictive capabilities of deep neural networks (DNNs) continue to evolve to increasingly impressive levels. However, it is still unclear how training procedures for DNNs succeed in finding parameters that produce good results for such high-dimensional and nonconvex loss functions. In particular, we wish to understand why simple optimization schemes, such as stochastic gradient descent, do not end up trapped in local minima with high loss values that would not yield useful predictions. We explain the optimizability of DNNs by characterizing the local minima and transition states of the loss-function landscape (LFL) along with their connectivity. We show that the LFL of a DNN in the shallow network or data-abundant limit is funneled, and thus easy to optimize. Crucially, in the opposite low-data/deep limit, although the number of minima increases, the landscape is characterized by many minima with similar loss values separated by low barriers. This organization is different from the hierarchical landscapes of structural glass formers and explains why minimization procedures commonly employed by the machine-learning community can navigate the LFL successfully and reach low-lying solutions.

Keywords: deep learning; energy landscapes; neural networks; optimization; statistical mechanics.

Fig. 1.

A DNN. Blue, input; red, output; green, hidden nodes.

Fig. 2.

Disconnectivity graphs for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in$ {100, 1,000, 2,000} training data (from top to bottom) for the DNNs with $H\in \left\{\mathrm{1,2,3}\right\}$ hidden layers (from left to right), labeled as “DATASET-#HL-#” on the top of each panel, where the two digits indicate, respectively, the number of hidden layers, $H$, and the amount of training data, $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$. Only the lowest 2,000 minima (or all, if fewer than 2,000 were identified) are shown, and the vertical scale has been adjusted to span the range of loss-function values within this set. Included as Insets below each disconnectivity graph are graphical visualizations of the performance of the global minimum (see SI Appendix, section S2 for details), as well as a plot of the training (horizontal axis) versus testing (vertical axis) loss values of all minima. It is apparent from these graphs that, in each case, the structure of the LFL is either funneled or comprises many minima with similar loss values connected by low barriers.

Fig. 3.

Continuation of Fig. 2 with $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in \left\{\mathrm{10,000,100,000}\right\}$.

Fig. 4.

Disconnectivity graphs for the training datasets OPTDIG (with $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in$ {1,500, 5,000} and $H\in \left\{\mathrm{1,3}\right\}$) and WINE (with $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ = 1,500 and $H\in \left\{\mathrm{1,3}\right\}$). Only the lowest 2,000 minima (or all of them if fewer than 2,000 were found) are shown. The vertical scale is adjusted to span the range of loss-function values within this set.