Maximizing the information learned from finite data selects a simple model

Abstract

We use the language of uninformative Bayesian prior choice to study the selection of appropriately simple effective models. We advocate for the prior which maximizes the mutual information between parameters and predictions, learning as much as possible from limited data. When many parameters are poorly constrained by the available data, we find that this prior puts weight only on boundaries of the parameter space. Thus, it selects a lower-dimensional effective theory in a principled way, ignoring irrelevant parameter directions. In the limit where there are sufficient data to tightly constrain any number of parameters, this reduces to the Jeffreys prior. However, we argue that this limit is pathological when applied to the hyperribbon parameter manifolds generic in science, because it leads to dramatic dependence on effects invisible to experiment.

Keywords: Bayesian prior choice; effective theory; information theory; model selection; renormalization group.

Fig. 1.

Optimal priors for the Bernoulli model (Eq. 1). Red lines indicate the positions of delta functions in $p_{\star }(\theta )$, which are at the maxima of $f_{\mathrm{KL}}(\theta )$, Eq. 3. As $m\to \mathrm{\infty }$ these coalesce into the Jeffreys prior $p_{\mathrm{J}}(\theta )$.

Fig. 2.

Convergence of the BA algorithm. This is for the one-parameter Gaussian model Eq. 2 with $L=10$ (comparable to $m=10$ in Fig. 1). Right shows $\theta$ discretized into 10 times as many points, but $p_{\tau }(\theta )$ clearly converges to the same 5 delta functions.

Fig. 3.

Behavior of $p_{\star }(\theta )$ with increasing Fisher length. A and B show the atoms of $p_{\star }(\theta )$ for the two one-dimensional models as $L$ is increased (i.e., we perform more repetitions $m$ or have smaller noise $\sigma$). C shows the scaling of the MI (in bits) with the number of atoms $K$. The dashed line is the bound $\mathrm{MI}\le \log K$, and the solid line is the scaling law $\mathrm{MI}\sim 3/4\log K$.

Fig. 4.

Parameters and priors for the exponential model (Eq. 5). A shows the area of the $\overrightarrow{y}$ plane covered by all decay constants $k_1,k_2\ge 0$. B shows the positions of the delta functions of the optimal prior $p_{\star }(\overrightarrow{y})$ for several values of $\sigma$, with colors indicating the dimensionality $r$ at each point. C shows the proportion of weight on these dimensionalities.

Fig. 5.

Distributions of expected data $p(x)$ from different priors. A is the one-parameter Gaussian model, with $L=10$. B projects the two-parameter exponential model onto the $y_1+y_2$ direction, for $\sigma =1/7$ where the perpendicular direction should be irrelevant. The length of the relevant direction is about the same as the one-parameter case: $L_{+}=7\sqrt{2}$. Note that the distribution of expected data $p(x_{+})$ from the Jeffreys prior here is quite different, with almost no weight at the ends of the range ($0$ and $\sqrt{2}$), because this prior still weights the area and not the length.