Dynamical systems learning by a circuit theoretic approach

In this paper, we derive a new general method for both on-line and off-line backward gradient computation of a system output, or cost function, with respect to system parameters, using a circuit theoretic approach. The system can be any causal, in general nonlinear and time-variant, dynamic system represented by a Signal Flow Graph (SFG), in particular any feedforward, time delay or recurrent neural network. The gradient is obtained in a straightforward way by the analysis of two numerical circuits, the original one and its adjoint (obtained from the first by simple transformations) without the complex chain rule expansions of derivatives usually employed.