Fast training of Recurrent Neural Networks by the Recursive Least Squares Method

In this work a novel approach to the training of recurrent neural nets is presented. the algorithm exploits the separability of each neuron into its linear and nonlinear part. Each iteration of the learning consists of two steps: first the descent of the error functional in the space of the linear outputs of the neurons is performed (descent in the neuron space); then the weights are updated by solving a linear system with a Recursive Least squares technique. The main properties of the new approach are high speed of convergence, favorable numerical conditioning and robustness. The numerical stability is assured by the use of robust LS linear system solvers, operating directly on the data.