Recurrent neural network architectures

In this chapter, we present three different recurrent neural network architectures that we employ for the prediction of real-valued time series. All the models reviewed in this chapter can be trained through the previously discussed backpropagation through time procedure. First, we present the most basic version of recurrent neural networks, called Elman recurrent neural network. Then, we introduce two popular gated architectures, which are long short-term memory and the gated recurrent units. We discuss the main advantages of these more sophisticated architectures, especially regarding their capability to process much longer dependencies in time by maintaining an internal memory for longer periods. For each one of the reviewed network, we provide the details and we show the equations for updating the internal state and computing the output at each time step. Then, for each recurrent neural network we also provide a quick overview of its main applications in previous works in the context of real-valued time series forecasting.