The Dropout Learning Algorithm

Abstract

Dropout is a recently introduced algorithm for training neural network by randomly dropping units during training to prevent their co-adaptation. A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates. The framework allows a complete analysis of the ensemble averaging properties of dropout in linear networks, which is useful to understand the non-linear case. The ensemble averaging properties of dropout in non-linear logistic networks result from three fundamental equations: (1) the approximation of the expectations of logistic functions by normalized geometric means, for which bounds and estimates are derived; (2) the algebraic equality between normalized geometric means of logistic functions with the logistic of the means, which mathematically characterizes logistic functions; and (3) the linearity of the means with respect to sums, as well as products of independent variables. The results are also extended to other classes of transfer functions, including rectified linear functions. Approximation errors tend to cancel each other and do not accumulate. Dropout can also be connected to stochastic neurons and used to predict firing rates, and to backpropagation by viewing the backward propagation as ensemble averaging in a dropout linear network. Moreover, the convergence properties of dropout can be understood in terms of stochastic gradient descent. Finally, for the regularization properties of dropout, the expectation of the dropout gradient is the gradient of the corresponding approximation ensemble, regularized by an adaptive weight decay term with a propensity for self-consistent variance minimization and sparse representations.

Keywords: backpropagation; ensemble; geometric mean; machine learning; neural networks; regularization; sparse representations; stochastic gradient descent; stochastic neurons; variance minimization.

Figure 1.1

Dropout training in a simple network. For each training example, feature detector units are dropped with probability 0.5. The weights are trained by backpropagation (BP) and shared with all the other examples.

Figure 1.2

Dropout prediction in a simple network. At prediction time, all the weights from the feature detectors to the output units are halved.

Figure 8.1

The curve associated with the approximate bound |E – NWGM| ≲ E(1 – E)|1 – 2E|/[1 – 2E(1 – E)] (Equation 87).

Figure 8.2

The curve associated with the approximate bound |E – NWGM| ≲ 2E(1 – E)|1 – 2E| (Equation 87).

Figure 9.1

Histogram of NWGM values for a random sample of 100 values O taken from: (1) the uniform distribution over [0,1] (upper left); (2) the uniform distribution over [0,0.5] (lower left); (3) the normal distribution with mean 0.5 and standard deviation 0.1 (upper right); and (4) the normal distribution with mean 0.25 and standard deviation 0.05 (lower right). All probability weights are equal to 1/100. Each sampling experiment is repeated 5,000 times to build the histogram.

Figure 9.2

Behavior of the Pearson correlation coefficient (left) and the covariance (right) between the empirical expectation E and the empirical NWGM as a function of the number of samples and sample distribution. For each number of samples, the sampling procedure is repeated 10,000 times to estimate the Pearson correlation and covariance. The distributions are the uniform distribution over [0,1], the uniform distribution over [0,0.5], the normal distribution with mean 0.5 and standard deviation 0.1, and the normal distribution with mean 0.25 and standard deviation 0.05.

Figure 9.3

Each row corresponds to a scatter plot for all the neurons in each one of the four hidden layers of a deep classifier trained on the MNIST dataset (see text) after learning. Scatter plots are derived by cumulating the results for 10 random chosen inputs. Dropout expectations are estimated using 10,000 dropout samples. The second order approximation in the left column (blue dots) correspond to |E – NWGM| ≈ V|1 – 2E|/(1 – 2V) (Equation 87). Bound 1 is the variance-dependent bound given by E(1 – E)|1 – 2E|/(1 – 2V) (Equation 87). Bound 2 is the variance-independent bound given by E(1–E)|1–2E|/(1–2E(1–E)) (Equation 87). In the right column, W represent the neuron activations in the deterministic ensemble network with the weights scaled appropriately and corresponding to the “propagated” NWGMs.

Figure 9.4

Similar to Figure 9.3, using the sharper but potentially more restricted second order approximation to the NWGM obtained by using a Taylor expansion around the mean (see Appendix B, Equation 202).

Figure 9.5

Similar to Figures 9.3 and 9.4. Approximation 1 corresponds to the second order Taylor approximation around 0.5: ∥E – NWGM| ≈ V|1 – 2E|/(1 – 2V) (Equation 87). Approximation 2 is the sharper but more restrictive second order Taylor approximation around $E:\frac{E-(V∕2E)}{1-(\left[0.5V\right]∕\left[E(1-E)\right])}$ (see Appendix B, Equation 202). Histograms for the two approximations are interleaved in each figure of the right column.

Figure 9.6

Empirical distribution of NWGM – E is approximately Gaussian at each layer, both before and after training. This was performed with Monte Carlo simulations over dropout subnetworks with 10,000 samples for each of 10 fixed inputs. After training, the distribution is slightly asymmetric because the activation of the neurons is asymmetric. The distribution in layer one before training is particularly tight simply because the input to the network (MNIST data) is relatively sparse.

Figure 9.7

Empirical distribution of W – E is approximately Gaussian at each layer, both before and after training. This was performed with Monte Carlo simulations over dropout subnetworks with 10,000 samples for each of 10 fixed inputs. After training, the distribution is slightly asymmetric because the activation of the neurons is asymmetric. The distribution in layer one before training is particularly tight simply because the input to the network (MNIST data) is relatively sparse.

Figure 9.8

Approximation of $E\left(O_i^l{O}_i^l\right)$ by $W_i^l$ and by $W_i^l{W}_i^l$ corresponding respectively to the estimates $W_i^l(1-W_i^l)$ and for the variance for neurons in a MNIST classifier network before and after training. Histograms are obtained by taking all non-input neurons and aggregating the results over 10 random input vectors.

Figure 9.9

Histogram of the difference between the dropout variance of $O_i^l$ and its approximate upperbound $W_i^l\left(W_i^l\right)$ in a MNIST classifier network before and after training. Histograms are obtained by taking all non-input neurons and aggregating the results over 10 random input vectors. Note that at the beginnning of learning, with random small weights, $E\left(O_i^l\right)\approx W_i^l\approx 0.5$, and thus $Var\left(O_i^l\right)\approx 0$ whereas $W_i^l(1-W_i^l)\approx 0.25$.

Figure 9.10

Temporal evolution of the dropout variance V(O) during training averaged over all hidden units.

Figure 9.11

Temporal evolution of the difference W(1 – W ) – V during training averaged over all hidden units.

Figure 9.12

Approximation of $E\left(O_i^l{O}_j^h\right)$ by $W_i^l{W}_j^h$ for pairs of non-input neurons that are not directly connected to each other in a MNIST classifier network, before and after training. Histograms are obtained by taking 100,000 pairs of unconnected neurons, uniformly at random, and aggregating the results over 10 random input vectors.

Figure 9.13

Comparison of $E\left(O_i^l{O}_j^h\right)$ to 0 for pairs of non-input neurons that are not directly connected to each other in a MNIST classifier network, before and after training. As shown in the previous figure, $W_i^l{W}_j^h$ provides a better approximation. Histograms are obtained by taking 100,000 pairs of unconnected neurons, uniformly at random, and aggregating the results over 10 random input vectors.

Figure 9.14

Approximation of $E\left(O_i^l{O}_j^h\right)$ by $W_i^l{W}_{ij}^{lh}$ and $W_i^l{W}_j^h$ for pairs of connected non-input neurons, with a directed connection from j to i in a MNIST classifier network, before and after training. Histograms are obtained by taking 100,000 pairs of connected neurons, uniformly at random, and aggregating the results over 10 random input vectors.

Figure 9.15

Histogram of the difference between E(σ′(S)) and σ′(E(S)) all non-input neurons, in a MNIST classifier network, before and after training. Histograms are obtained by taking all non-input neurons and aggregating the results over 10 random input vectors. The nodes in the first hidden layer have 784 sparse inputs, while the nodes in the upper three hidden layers have 1200 non-sparse inputs. The distribution of the initial weights are also slightly different for the first hidden layer. The differences between the first hidden layer and all the other hidden layers are responsible for the initial bimodal distribution.

Figure 10.1

A spiking neuron formally operates in 3 steps by computing first a linear sum S, then a probability O = σ(S), then a stochastic output Δ of size r with probability O(and 0 otherwise).

Figure 10.2

Three closely related networks. The first network operates stochastically and consists of spiking neurons: a neuron sends a spike of size r with probability O. The second network operates stochastically and consists of logistic dropout neurons: a neurons sends an activation O with a dropout probability r. The connection weights in the first and second networks are identical. The third network operates in a deterministic way and consists of logistic neurons. Its weights are equal to the weights of the second network multiplied by the corresponding probability r.

Figure 11.1

Empirical distribution of final neuron activations in each layer of the trained MNIST classifer demonstrating the sparsity. The empirical distributions are combined over 1000 different input examples.

Figure 11.2

The three phases of learning. For a particular input, a typical active neuron (red) starts out with low dropout variance, experiences an increase in variance during learning, and eventually settles to some steady constant consitency value. A typical inactive neuron (blue) quickly learns to stay silent. Its dropout variance grows only minimally from the low initial value. Curves correspond to mean activation with 5% and 95% percentiles. This is for a single fixed input, and 1000 dropout Monte Carlo simulations.

Figure 11.3

Consistency of active neurons does not noticeably decline in the upper layers. ’Active’ neurons are defined as those with activation greater than 0.1 at the end of training. There were at least 100 active neurons in each layer. For these neurons, 1000 dropout simulations were performed at each time step of 100 training epochs. The plot represents the dropout mean standard deviation and 5%, 95% percentiles computed over all the active neurons in each layer. Note that the standard deviation does not increase for the higher layers.