Information theoretical approach to the storage capacity of neural networks with binary weights

Abstract

The storage capacity of the perceptron with binary weights w(i)in[0,1] is derived by introducing the minimum distance d between input patterns. The approach presented in this paper is based on some results in the information theory, and the obtained storage capacity 0.585 is in good agreement with the well-known value 0.59 by the replica method in statistical physics. A strength of the present information theoretical approach is that it provides an easier and more intuitive understanding for the storage capacity than the replica method, which is believed to be more reliable and informative than the Vapnik-Chervonenkis procedure.