Aitken-based acceleration methods for assessing convergence of multilayer neural networks

Abstract

This paper first develops the ideas of Aitken delta(2) method to accelerate the rate of convergence of an error sequence (value of the objective function at each step) obtained by training a neural network with a sigmoidal activation function via the backpropagation algorithm. The Aitken method is exact when the error sequence is exactly geometric. However, theoretical and empirical evidence suggests that the best possible rate of convergence obtainable for such an error sequence is log-geometric. This paper develops a new invariant extended-Aitken acceleration method for accelerating log-geometric sequences. The resulting accelerated sequence enables one to predict the final value of the error function. These predictions can in turn be used to assess the distance between the current and final solution and thereby provides a stopping criterion for a desired accuracy. Each of the techniques described is applicable to a wide range of problems. The invariant extended-Aitken acceleration approach shows improved acceleration as well as outstanding prediction of the final error in the practical problems considered.