How does an LSTM cell solve the vanishing gradient problem?

Question

AcadiFi · Accepted Answer

Long Short-Term Memory (LSTM), introduced by Hochreiter and Schmidhuber (1997), augments the RNN cell with three gates and a cell state C_t that acts as an information highway.

Gates at time t (all sigmoid-activated, output in [0,1]):
- **Forget gate** f_t = sigma(W_f [h_{t-1}, x_t] + b_f): decides what to erase from cell state.
- **Input gate** i_t = sigma(W_i [h_{t-1}, x_t] + b_i): decides what to write.
- **Output gate** o_t = sigma(W_o [h_{t-1}, x_t] + b_o): decides what to expose as hidden state.

Candidate values: C_tilde_t = tanh(W_c [h_{t-1}, x_t] + b_c)
Cell state update: C_t = f_t * C_{t-1} + i_t * C_tilde_t
Hidden state: h_t = o_t * tanh(C_t)

Key insight: the additive update C_t = f_t * C_{t-1} + ... means gradients flow through the cell state without repeated multiplication by small derivatives. If f_t stays near 1, the cell state preserves information over hundreds of time steps.

Finance use case: Credit default prediction using 24 months of borrower transaction data. A vanilla RNN struggles to connect a Month-1 event (job loss) to a Month-22 outcome (default). An LSTM can hold that context in its cell state across all 24 months via a forget gate that rarely triggers on that specific pathway.

```mermaid
graph LR
  A[x_t, h_t-1] --> F[Forget gate]
  A --> I[Input gate]
  A --> O[Output gate]
  A --> C[Candidate]
  F --> S[Cell state C_t]
  I --> S
  C --> S
  S --> H[Hidden h_t]
  O --> H
```

Parameter count: 4 x (n_h x (n_x + n_h) + n_h) for a single layer, which is four times a vanilla RNN.

Variants: bidirectional LSTM (reads sequence forwards and backwards), stacked LSTM (multiple layers), attention-augmented LSTM.

How does an LSTM cell solve the vanishing gradient problem?

Master Level II with our CFA Course

Related Questions

Practice Questions