An Elementary Introduction to Information Geometry

Abstract

In this survey, we describe the fundamental differential-geometric structures of information manifolds, state the fundamental theorem of information geometry, and illustrate some use cases of these information manifolds in information sciences. The exposition is self-contained by concisely introducing the necessary concepts of differential geometry. Proofs are omitted for brevity.

Keywords: Bayesian hypothesis testing; Fisher–Rao distance; Hessian manifolds; affine connection; conjugate connections; curvature and flatness; differential geometry; dual metric-compatible parallel transport; dually flat manifolds; exponential family; gauge freedom; information manifold; metric compatibility; metric tensor; mixed parameterization; mixture clustering; mixture family; parameter divergence; separable divergence; statistical divergence; statistical invariance; statistical manifold; α-embeddings.

Figure A1

Illustration of the chordal slope lemma.

Figure 1

The parameter inference $\widehat{\theta }$ of a model from data $\mathcal{D}$ can also be interpreted as a decision making problem: decide which parameter of a parametric family of models $M={\left\{m_{\theta }\right\}}_{\theta \in \mathsf{\Theta }}$ suits the “best” the data. Information geometry provides a differential-geometric structure on manifold M which useful for designing and studying statistical decision rules.

Figure 2

Primal basis (red) and reciprocal basis (blue) of an inner product $\langle ·,·\rangle$ space. The primal/reciprocal basis are mutually orthogonal: $e^1$ is orthogonal to $e_2$, and $e_1$ is orthogonal to $e^2$.

Figure 3

Illustration of the parallel transport of vectors on tangent planes along a smooth curve. For a smooth curve c, with $c\left(0\right)=p$ and $c\left(1\right)=q$, a vector $v_p\in T_p$ is parallel transported smoothly to a vector $v_q\in T_q$ such that for any $t\in [0,1]$, we have $v_{c\left(t\right)}\in T_{c\left(t\right)}$.

Figure 4

Parallel transport with respect to the metric connection: the curvature effect can be visualized as the angle defect along the parallel transport on smooth (infinitesimal) loops. For a sphere manifold, a vector parallel-transported along a loop does not coincide with itself (e.g., a sphere), while it always conside with itself for a (flat) manifold (e.g., a cylinder).

Figure 5

Differential-geometric concepts associated to an affine connection ∇ and a metric tensor g.

Figure 6

Dual Pythagorean theorems in a dually flat space.

Figure 7

Five concentric pairs of dual Itakura–Saito circles.

Figure 8

Common dually flat spaces associated to smooth and strictly convex generators.

Figure 9

Visualizing the Cramér–Rao lower bound: the red ellipses display the Fisher information matrix of normal distributions $N(\mu ,{\sigma }^2)$ at grid locations. The black ellipses are sample covariance matrices centered at the sample means calculated by repeating 200 runs of sampling 100 iid variates for the normal parameters of the grid.

Figure 10

A divergence satisfies the property of information monotonicity iff $D({\theta }_{\overline{\mathcal{A}}}:{\theta }_{\overline{\mathcal{A}}}^{\prime })\le D(\theta :{\theta }^{\prime })$. Here, parameter $\theta$ represents a discrete distribution.

Figure 11

Overview of the main types of information manifolds with their relationships in information geometry.

Figure 12

Statistical Bayesian hypothesis testing: the best Maximum A Posteriori (MAP) rule chooses to classify an observation from the class that yields the maximum likelihood.

Figure 13

Exact geometric characterization (not necessarily i closed-form) of the best exponent error rate ${\alpha }^{*}$.

Figure 14

Geometric characterization of the best exponent error rate in the multiple hypothesis testing case.

Figure 15

Example of a mixture family of order $D=2$ (3 components: Laplacian, Gaussian and Cauchy prefixed distributions).

Figure 16

Example of w-GMM clustering into $k=2$ clusters.

Figure 17

Principled classes of distances/divergences.