New second-order algorithms for recurrent neural networks based on conjugate gradient

We derive two second-order algorithms, based on the conjugate gradient method, for online training of recurrent neural networks. These algorithms use two different techniques to extract second-order information on the Hessian matrix without calculating or storing it and without making numerical approximations. Several simulation results for nonlinear system identification tests by locally recurrent neural networks are reported for both the off-line and online case.