Recurrent Networks: LSTM & GRU

5.1 Why order matters

A time series carries meaning in its sequence: "clouds thickening over the last hour" and "clouds clearing over the last hour" can contain the same values in reverse order but imply opposite futures. A feedforward network cannot see that distinction structurally. Paper 1's FFNN receives the input window in exactly this flattened way: the entire sliding-window sequence of feature vectors is flattened into a single vector — hour-by-hour structure collapsed into one long list of numbers. The network can still learn temporal patterns from such a vector, but nothing in its architecture knows that entry 47 came just after entry 46.

Recurrent neural networks (RNNs) take the opposite approach: read the window one time-step at a time, in order, carrying a memory forward. Order is built into the machine, not left for it to infer.

5.2 Recurrent neural networks

An RNN keeps a hidden state h_t — a vector that acts as a running summary of everything read so far. At each time-step it combines the previous summary with the new input:

Crucially, the same weights are used at every step — one small machine applied repeatedly, like a single reader moving along a sentence, rather than a separate network per time-step. To picture training, unroll the loop: an RNN processing a 36-step window behaves like a 36-layer-deep network in which every layer shares the same weights, each layer consuming one time-step. That picture makes the next section's problem obvious.

5.3 Vanishing gradients

Training means backpropagating error through that unrolled chain. The chain rule multiplies local slopes, step after step. If those slopes are typically a little below 1, their product shrinks exponentially with sequence length:

The gradient reaching early time-steps becomes vanishingly small — so the network stops learning from the distant past, exactly the long-range structure we wanted it to capture. This is the vanishing gradient problem. Its mirror image, exploding gradients (slopes above 1 multiplying into huge, destabilizing updates), is the other failure mode. Together they explain why plain RNNs struggle with long sequences — and they are the direct motivation for the gated architectures of the next two sections.

5.4 LSTM: memory with gates

The Long Short-Term Memory network adds a second carried vector: the cell state — picture a conveyor belt running the length of the sequence, on which information rides largely undisturbed. Three sigmoid gates (each a small learned layer outputting values between 0 and 1, i.e. "how much to let through") regulate the belt:

• Forget gate — what to erase from the cell state (yesterday's cloud front no longer matters — drop it).
• Input gate — what new information to write onto the cell state (irradiance just plunged — record it).
• Output gate — what part of the cell state to reveal as this step's hidden state and output.

The decisive design point: the cell state is updated additively — new information is added to the belt rather than the whole memory being squashed through a transformation at every step. Additive updates give gradients a protected path backward through time, so error signals survive long spans instead of vanishing. That is why LSTMs can exploit long histories.

5.5 GRU: the streamlined cousin

The Gated Recurrent Unit asks: do we need all that machinery? It merges the cell state and hidden state into one vector and manages it with just two gates:

• Update gate — how much of the old state to keep versus replace with the new candidate (roughly, forget-gate and input-gate fused into one dial).
• Reset gate — how much of the old state to consult when proposing the new candidate.

Fewer gates and one state vector mean fewer parameters and cheaper training — and in practice GRUs often perform comparably to LSTMs, sometimes better, especially on smaller datasets where the LSTM's extra capacity is more burden than blessing.

5.6 Sequence-to-vector and multi-target output

How does a step-by-step reader emit 21 forecast values? Paper 1's RNNs use a many-to-one (sequence-to-vector) architecture: the window's feature vectors are fed in sequentially, one per time-step; only the final state — the summary after the whole window is read — is passed to a fully connected output layer with ReLU activation, which emits all 21 forecast values at once (multi-target regression, with ReLU conveniently clamping any negative power predictions to zero).

Contrast the FFNN's route to the same 21 outputs:

Same input data, same output shape, same loss — the only real difference is how the window is digested: sequentially with memory, or all at once as a flat vector.

5.7 FFNN vs RNN: what the results teach

The chosen window lengths were a character study of the three models: the FFNN did best with just 1 h of history, while the LSTM preferred 3 h and the GRU 6 h. Paper 1 attributes this to "the ability of the RNN-based models to prioritise historic input data, without being overwhelmed by the number of time-steps received as model input" — the recurrent models could take in long histories and gate away what didn't matter, whereas feeding the FFNN a 6-hour flattened vector just made its input unwieldy.

But the headline is how small the differences stayed:

• The LSTM beat the FFNN on clear days — yet lost on clear-intermittent days (FFNN MAPE 2.24% vs LSTM 2.45%).
• On average, the FFNN even outperformed the LSTM (macro NRMSE 8.19% vs 8.23%).
• The GRU was best overall at 8.12% — the streamlined recurrent model, not the heavyweight one.