Information Entropy of Biometric Data in a Recurrent Neural Network with Low Connectivity

Abstract

In this paper, we explore the storage capacity and maximal information content of a random recurrent neural network characterized by a very low connectivity. A specific set of patterns is embedded into the network according to the Hebb prescription, a fundamental principle in neural learning. We thoroughly examine how various properties of the network, such as its connectivity and the level of synaptic noise, influence its performance and information retention capabilities, which is evaluated through an entropy measure. Our theoretical analyses are complemented by extensive simulations, and the results are validated through comparisons with the retrieval of real biometric patterns, including retinal vessel maps and fingerprints. This comprehensive approach provides deeper insights into the functionality and limitations of finite-connectivity neural networks and their applicability to the retrieval of complex, structured patterns.

Keywords: disordered systems; neural networks; storage capacity; synaptic dilution.

Figure 1

Finite Connectivity Attractor Neural Network (FCANN) schematic. Hebbian learning of random and biometric patterns (fingerprints and retinal vessels).

Figure 2

Retrieval overlap m, spin-glass parameter q (solid lines) and overlap between replicas $q^{\prime }$ (dashed lines) versus temperature T. (a) $c=5$, $p=2$ (black), $p=3$ (red) and $p=4$ (green); (b) $c=20$, $p=5$ (black), $p=13$ (red) and $p=17$ (green). The RS solution becomes unstable at $T<T_{AT}$, pointed out by the bifurcation between q and $q^{\prime }$. Dotted lines in the upper graphics represent simulation results for a network with $N=100,000$ neurons. In the lower graphics, the dashed lines represent the $q^{\prime }$.

Figure 3

Phase diagram T versus $\alpha =p/c$ for $c=20$ (black) and $c=10$ (red) and $c=5$ (green). Symbols indicate the outcome of calculations, and lines are only to guide the eyes. For comparison, results for the extremely diluted $c\to \infty$ are also shown (blue), from ref. [24].

Figure 4

Information content as a function of the connectivity, at $T=0$, for several values of p, RS results. Dots: corresponding simulation results for a network with $N=$ 100,000 neurons. The results show that for each number of patterns there is a connectivity that maximizes the information content. Furthermore, the maximal information content is obtained with a low number of patterns. This should be a valuable information for designers of artificial neural networks.

Figure 5

(a) Maximum information content $i_{\max }$ as a function of p, at $T=0$ (black), $T=0.1$ (red) $T=0.2$ (green), $T=0.4$ (blue) and $T=0.6$ (orange). (b) The corresponding ${\alpha }_{\max }=p/c_{\max }$, where $c_{\max }$ is the connectivity value that maximizes the information content, for each p value. The same color scheme as in (a). The lines are only to guide the eyes.

Figure 6

Retinal vessel patterns retrieval at $T=0$. The patterns are binarized images of retinas, consisting of $N=316\times 316=$ 99,856 pixels and $c=10$.

Figure 7

Fingerprint patterns retrieval at $T=0$. The patterns are binarized images of fingerprints, consisting of $N=340\times 263=$ 89,420 pixels and $c=10$.