Other recurrent neural networks models

In this chapter we review two additional types of Recurrent Neural Network, which present important differences with respect to the architectures described so far. More specifically, we introduce the nonlinear auto-regressive with eXogenous inputs (NARX) neural network and the Echo State Network. Both these networks have been largely employed in Short Term Load Forecast applications and they have been shown to be more effective than other methods based on statistical models. The main differences of NARX networks and Echo State Networks with respect to the other previously described models, are both in terms of their architecture and, in particular, in their training procedure. Indeed, both these architectures are designed in such a way that Back Propagation Through Time is not necessary. Specifically, in NARX the network output is replaced by the expected ground truth and this allows to train the network like a feedforward architecture. On the other hand, in a Echo State Network only the outermost linear layer is trained, usually by means of ridge regression. Due to these fundamental differences, some of the properties and training approaches discussed in the previous sections do not hold for the NARX and Echo State Network models and we reserved a separate chapter to review these models.