How does a recurrent neural network differ from a feedforward network?

Question

AcadiFi · Accepted Answer

A recurrent neural network processes sequences one element at a time while carrying a hidden state that summarizes past inputs. At each step t:
h_t = f(W_x x_t + W_h h_{t-1} + b)
y_t = g(W_y h_t + c)

The same weights W_x, W_h, W_y are reused at every time step (weight sharing across time), so the network can process arbitrarily long sequences with a fixed parameter count.

Finance example: Forecasting next day's S&P return from a 20-day rolling input of (return, volume, VIX). A feedforward network sees the 20 days as a flat vector of 60 features - it cannot easily encode 'recent values matter more.' An RNN walks through the 20 days, updating its hidden state, and emits a prediction after the 20th step. The recurrence inherently encodes temporal order.

Training uses backpropagation through time (BPTT): unroll the network across the sequence length, compute gradients at each step, sum them for shared weights.

Problems with vanilla RNN:
- **Vanishing gradients** over long sequences: multiplying many small Jacobians squashes distant-past information.
- **Exploding gradients**: large Jacobians cause instability (can be mitigated with clipping).
- **Short memory**: in practice, vanilla RNN struggles beyond ~10-20 time steps.

Solutions: LSTM and GRU (see related questions), which use gating mechanisms to control information flow.

```mermaid
graph LR
  X1[x_t-2] --> H1[h_t-2]
  H1 --> H2[h_t-1]
  X2[x_t-1] --> H2
  H2 --> H3[h_t]
  X3[x_t] --> H3
  H3 --> Y[y_t]
```

CFA context: know the recurrent loop, weight sharing across time, and why RNNs suit time-series and NLP better than vanilla feedforward networks.

How does a recurrent neural network differ from a feedforward network?

Master Level II with our CFA Course

Related Questions

Practice Questions