Recurrent networks and LSTMs

In short

A recurrent neural network (RNN) processes a sequence one element at a time while carrying a hidden state that summarises everything seen so far, so the same weights are reused at every time step. This weight sharing lets one model handle variable-length input — text, speech, sensor streams, prices — but training by backpropagation through time (BPTT) makes gradients vanish or explode over long spans. The LSTM (Hochreiter and Schmidhuber, 1997) and the simpler GRU (2014) solve this with gated additive memory, and remain the standard answer for long-range dependence in sequences. Universities examine the recurrence equations, unrolling, BPTT derivation, the vanishing-gradient argument and LSTM gate arithmetic; GATE DA/CS asks parameter counts, output shapes and gradient-decay reasoning. Indian employers (Jio, Bhashini-linked labs, Zoho, Freshworks, Flipkart, Swiggy, HDFC/ICICI analytics, Tata Elxsi, Qualcomm, Honeywell, ISRO/IMD groups) still use RNNs and LSTMs in speech, forecasting, fraud and edge deployments where Transformers are too heavy.

Explained

Why recurrence, and the basic cell

• A feed-forward net needs a fixed-size input and treats position 1 and position 50 with different weights; a sequence has variable length and shift-invariant structure.
• An RNN defines a recurrence: h_t = f(x_t, h_{t-1}). The hidden state h_t is a learned, lossy summary of x_1…x_t — it is the network's memory.
• Parameter sharing: the same W_xh, W_hh, b_h act at every step, so the model generalises across positions and the parameter count does not grow with sequence length T.
• Elman network (the standard "simple RNN"): recurrence through the hidden layer. Jordan network: recurrence from the *output* back to the hidden layer.
• Unrolling (unfolding): drawing the cell T times converts the cyclic graph into a deep feed-forward graph of depth T with tied weights — this is what makes backpropagation possible.
• tanh is the usual hidden activation (zero-centred, output in (−1, 1)); ReLU in a recurrent loop easily diverges because the state is multiplied repeatedly.

Input–output patterns (classification of RNN architectures)

1. One-to-one — ordinary net, no recurrence needed.
2. One-to-many — image captioning, music generation: one input, a generated sequence.
3. Many-to-one — sentiment classification, ECG arrhythmia detection: read the whole sequence, output once (usually from h_T or a mean/max pool over all h_t).
4. Many-to-many, synchronised (T_in = T_out) — POS tagging, named-entity recognition, per-frame labelling.
5. Many-to-many, asynchronous (encoder–decoder / seq2seq) — translation, summarisation, speech-to-text: an encoder compresses the input to a context vector, a decoder emits the output sequence.
6. Bidirectional RNN (BiRNN/BiLSTM): one pass left-to-right, one right-to-left, states concatenated — doubles parameters, gives both past and future context. Valid only when the whole sequence is available (not for real-time/streaming or autoregressive generation).
7. Stacked/deep RNN: layer 1's output sequence is layer 2's input; 2–4 layers typical. Depth in *features* is separate from depth in *time*.

BPTT — backpropagation through time

• Total loss over a sequence: L = Σ_t L_t(ŷ_t, y_t). Gradients flow backwards through the unrolled graph.
• Because W_hh is shared, its gradient accumulates contributions from every time step: ∂L/∂W_hh = Σ_t ∂L_t/∂W_hh, and each term itself sums over all earlier steps k ≤ t.
• Derivation outline (examined): write ∂L_t/∂W_hh = Σ_{k=1..t} (∂L_t/∂h_t)(∂h_t/∂h_k)(∂h_k/∂W_hh), where the Jacobian product ∂h_t/∂h_k = Π_{j=k+1..t} W_hh^T diag(f'(z_j)).
• Truncated BPTT (TBPTT): carry the hidden state forward but backpropagate only k steps (typically 20–200). Cuts memory from O(T) to O(k); the cost is that dependencies longer than k are never learnt.
• Memory cost: all T hidden activations must be stored for the backward pass — the usual cause of GPU OOM on long sequences.
• RNN training is inherently sequential in time (step t needs h_{t-1}), which is why RNNs do not parallelise across the time axis the way CNNs or Transformers do.

Vanishing and exploding gradients

• The Jacobian product is a product of t − k matrices. If the relevant singular value of W_hh diag(f') is < 1 the gradient decays geometrically (vanishing); if > 1 it grows geometrically (exploding).
• Saturating activations make it worse: tanh'(z) ≤ 1 and σ'(z) ≤ 0.25, so the factor is usually below 1 even for moderate weights.
• Symptoms: vanishing → loss plateaus, the model learns only short-range patterns, early-step gradients ≈ 0. Exploding → loss spikes to NaN, weights blow up in one update.
• Remedies: gradient clipping (for exploding), gated cells (for vanishing), orthogonal/identity initialisation of W_hh, layer normalisation, skip connections, shorter effective horizons, and careful learning rates.
• Note the asymmetry: clipping fixes explosion cheaply, but *no* rescaling fixes vanishing — the information is genuinely lost, hence the need for an architectural fix.

LSTM — the gated cell

• Adds a second state, the cell state c_t, updated *additively*. The path c_{t-1} → c_t has no repeated matrix multiplication — the constant error carousel — so gradients survive hundreds of steps.
• Three sigmoid gates, each a learned soft switch in (0, 1), computed from [x_t, h_{t-1}]:
  • **Forget gate f_t** — how much of the old cell content to keep (f=1 keep all, f=0 erase).
  • **Input gate i_t** — how much of the new candidate g_t = tanh(...) to write.
  • **Output gate o_t** — how much of the cell to expose as the hidden state h_t = o_t ⊙ tanh(c_t).
• c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t: the gradient of c_t w.r.t. c_{t-1} is f_t, elementwise — so if the forget gate stays near 1 the gradient is carried almost unchanged.
• Forget-gate bias initialised to 1 (or 2) is standard practice: it starts the cell in "remember" mode and markedly speeds early training.
• Variants: *peephole* LSTM (gates also see c_{t-1}); *coupled input–forget* (i_t = 1 − f_t); *GRU*, which merges cell and hidden state and uses two gates — update z_t and reset r_t — with about 75% of an LSTM's parameters.
• LSTM vs GRU in practice: comparable accuracy on most tasks; GRU trains faster and suits small data, LSTM often edges ahead on very long sequences and large-corpus language modelling. Treat it as a hyperparameter, not a doctrine.

Training recurrent models in practice

• Padding and masking: batch sequences of unequal length by padding to the longest, then mask padded steps out of the loss; PyTorch's pack_padded_sequence skips them entirely. Bucketing groups similar lengths to cut wasted compute.
• Teacher forcing: during training the decoder is fed the ground-truth previous token, not its own prediction — fast and stable, but creates exposure bias at inference. Scheduled sampling mixes in model predictions to reduce it.
• Regularisation: dropout on the input/output connections of each layer; *variational (locked) dropout* uses the same mask at every time step — applying a fresh mask per step to the recurrent connection destroys memory. Zoneout randomly keeps the previous state instead of zeroing it. Also weight decay, and weight tying between input embedding and output softmax in language models.
• Normalisation: batch norm works poorly across time (statistics differ per step); use layer normalisation inside the recurrence.
• Embeddings: discrete tokens enter through a learned embedding matrix (vocab × d). Output is a softmax over the vocabulary; large vocabularies use sampled softmax or adaptive softmax.
• Decoding: greedy, or beam search (beam 3–10) with length normalisation; CTC loss for unsegmented speech/handwriting where input and label lengths differ and no alignment is given.
• Optimisers: Adam with gradient clipping is the default; SGD with momentum plus clipping is still competitive for language models.
• Where RNNs still win: streaming/low-latency inference (O(1) memory per step, no growing KV cache), very long or unbounded streams, tiny on-device models, and small datasets. Transformers dominate when sequences are moderate and parallel training throughput matters.

Key points

• Recurrence — h_t = tanh(W_xh x_t + W_hh h_{t-1} + b_h); one weight set reused at every step.
• Unrolling — turns the cycle into a depth-T tied-weight feed-forward graph for backprop.
• Weight sharing — parameter count independent of sequence length; enables variable-length input.
• BPTT — gradients summed over all time steps because W_hh is shared.
• Truncated BPTT — backpropagate k steps only; bounds memory, caps learnable dependency length.
• Vanishing gradient — Jacobian product decays geometrically when spectral factor < 1; blocks long-range learning.
• Exploding gradient — same product with factor > 1; fixed by clipping the global gradient norm.
• Cell state — the LSTM's additive memory highway; its gradient multiplier is the forget gate, not W_hh.
• Forget / input / output gates — sigmoid switches deciding erase, write and read of memory.
• Constant error carousel — the near-identity c_{t-1} → c_t path that preserves gradient.
• Forget-bias = 1 — standard initialisation; starts the cell remembering.
• GRU — update and reset gates, no separate cell state, ~3/4 of LSTM parameters.
• Bidirectional — concatenates forward and backward states; unusable for streaming or generation.
• Teacher forcing / exposure bias — training on ground truth, testing on own outputs.
• CTC — loss for unaligned sequence labelling (speech, OCR).
• Masking — padded steps must be excluded from loss and state updates.
• Perplexity — exp(cross-entropy in nats); the standard language-model metric.
• **Orthogonal init of W_hh** — keeps singular values at 1, delaying vanishing/explosion.

Formulas and examples

Example 1 — Parameter count (GATE style). Input m = 100, hidden n = 128, output classes 10.

• Simple RNN: W_xh = 100×128 = 12,800; W_hh = 128×128 = 16,384; b_h = 128 → 29,312. Output layer 128×10 + 10 = 1,290. Total = 30,602.
• LSTM: 4 × 29,312 = 117,248 recurrent parameters, + 1,290 = 118,538.
• GRU: 3 × 29,312 = 87,936, + 1,290 = 89,226.
• A BiLSTM with the same sizes: 2 × 117,248 = 234,496 recurrent; the classifier now sees 256 features → 256×10 + 10 = 2,570; total 237,066.

Example 2 — Vanishing vs exploding, quantified. Suppose the per-step gradient factor ‖W_hh diag(tanh′)‖ is effectively a constant λ, over 30 steps.

• λ = 0.6: 0.6³⁰ = (0.6¹⁰)³ = (6.047×10⁻³)³ ≈ 2.2 × 10⁻⁷ — the step-1 gradient is seven orders of magnitude smaller than the step-30 gradient; nothing is learnt at the start of the sequence.
• λ = 1.2: 1.2³⁰ = (1.2¹⁰)³ = 6.1917³ ≈ 237 — a 237× blow-up, giving loss spikes and NaNs.
• An LSTM with forget gates at f = 0.99: 0.99³⁰ ≈ 0.74 — only a 26% decay over 30 steps. That single contrast is the whole justification for the LSTM.

Example 3 — One LSTM step by hand (scalar cell, x_t = 1.0, h_{t−1} = 0.5, c_{t−1} = 0.8).

• c_t = 0.7773 × 0.8 + 0.6682 × 0.7616 = 0.6219 + 0.5090 = **1.1309**
• tanh(1.1309) = 0.8113; h_t = 0.6792 × 0.8113 = **0.5510**
• Note |c_t| > 1 is perfectly legal — the cell is unbounded; only h_t is squashed into (−1, 1).

Example 4 — Gradient clipping arithmetic. Global gradient norm ‖g‖₂ = 25, threshold τ = 5. Scale = 5/25 = 0.2, so every component is multiplied by 0.2 and the clipped norm is exactly 5. The *direction* is preserved; only the magnitude changes — which is why clipping does not bias the update.

Example 5 — Perplexity and memory. A Hindi–English code-mixed LSTM language model reaches validation cross-entropy 4.50 nats/token.

• PPL = e^{4.50} ≈ 90.0 — the model is as uncertain as if choosing uniformly among 90 words. After adding a second layer the loss falls to 4.10 → PPL = e^{4.10} ≈ 60.3, a 33% reduction in perplexity.
• Activation memory for BPTT with batch 32, T = 400, n = 512, LSTM (store h, c and 4 gate pre-activations ≈ 6 tensors), float32: 32 × 400 × 512 × 6 × 4 bytes ≈ 157 MB for one layer — hence truncation to k = 100 cuts it to ~39 MB.

import torch, torch.nn as nn
class Tagger(nn.Module):
    def __init__(self, vocab, emb=128, hid=256, tags=12):
        super().__init__()
        self.emb  = nn.Embedding(vocab, emb, padding_idx=0)
        self.lstm = nn.LSTM(emb, hid, num_layers=2, batch_first=True,
                            bidirectional=True, dropout=0.3)
        self.out  = nn.Linear(2 * hid, tags)
    def forward(self, x, lengths):
        p = nn.utils.rnn.pack_padded_sequence(
                self.emb(x), lengths.cpu(), batch_first=True, enforce_sorted=False)
        o, _ = self.lstm(p)
        o, _ = nn.utils.rnn.pad_packed_sequence(o, batch_first=True)
        return self.out(o)                      # (B, T, tags)

# loss ignores padding; clip before every optimiser step
crit = nn.CrossEntropyLoss(ignore_index=0)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)

Where it is used

• Indian-language speech and text: acoustic models and sequence taggers for the Bhashini / IndicNLP ecosystem, IIT and IIIT speech labs, and voice IVR at banks and telecoms; BiLSTM-CRF is still a strong baseline for Indic NER and POS where labelled data is scarce.
• Banking and fintech: card and UPI fraud detection over a customer's transaction sequence, loan-default early-warning from repayment history, and NSE/BSE and commodity forecasting at quant desks — audit trails and model governance follow RBI model-risk expectations, and customer sequence data is personal data under the Digital Personal Data Protection Act, 2023.
• Demand and operations forecasting: SKU-level demand at Flipkart/Reliance Retail, food-delivery and ride ETA prediction at Swiggy/Zomato/Ola, electricity load forecasting for SLDCs and discoms, IMD/hydrology inflow prediction.
• Industrial and embedded: predictive maintenance from vibration and current signatures (Tata Steel, L&T, Honeywell, Siemens India), handwriting/OCR for forms and cheques, ECG and EEG classification in medical-device startups; small LSTMs and GRUs run on microcontrollers via TensorFlow Lite Micro where a Transformer will not fit.
• Job roles: ML engineer, data scientist (time series), speech/NLP engineer, quant analyst, edge-AI engineer — interviews here routinely ask for the LSTM equations and the vanishing-gradient argument from memory.

Interview questions

• Q: Why can an RNN handle variable-length input when a feed-forward net cannot? A: It applies the same weights at every time step and carries a fixed-size hidden state, so the number of parameters is independent of sequence length.
• Q: What exactly vanishes in the vanishing-gradient problem? A: The Jacobian product ∂h_t/∂h_k, a product of t−k matrices each with factor typically below 1, so gradients reaching early time steps decay geometrically to near zero.
• Q: How does the LSTM fix it? A: The cell state is updated additively and its gradient multiplier is the forget gate, so with f ≈ 1 the gradient passes through many steps almost unchanged — no repeated weight matrix multiplication.
• Q: Why clip gradients instead of lowering the learning rate? A: Explosions are rare spikes; clipping caps only those steps while preserving direction, whereas a globally tiny learning rate slows all normal training.
• Q: Difference between LSTM and GRU? A: GRU merges cell and hidden state and uses two gates (update, reset) instead of three, so it has about 25% fewer parameters and trains faster, with broadly similar accuracy.
• Q: When must you not use a bidirectional LSTM? A: When future input is unavailable — real-time streaming, online speech recognition, or any autoregressive generation — because the backward pass needs the whole sequence.
• Q: What is teacher forcing and what does it cost you? A: Feeding the decoder ground-truth previous tokens during training; it stabilises and speeds learning but causes exposure bias, since at inference the model must consume its own possibly wrong outputs.
• Q: Why initialise the forget-gate bias to 1? A: It makes the gate start near 1 so the cell retains information from the first epochs, instead of the model having to learn not to erase its memory.
• Q: How many parameters does an LSTM with input 64 and hidden 128 have? A: 4 × 128 × (64 + 128 + 1) = 98,816.
• Q: Why does dropout on recurrent connections need the same mask at every step? A: A fresh mask per step injects independent noise into the memory path and destroys the state; a locked mask regularises without erasing long-range information.
• Q: Why are RNNs slower to train than Transformers on the same data? A: The recurrence is sequential in time — step t needs h_{t−1} — so the time axis cannot be parallelised, while attention processes all positions at once.
• Q: Why use CTC loss in speech recognition? A: Input frames vastly outnumber output characters and no alignment is labelled; CTC marginalises over all valid alignments using a blank symbol.

Common mistakes

• Writing the LSTM as c_t = f_t·c_{t−1} + i_t·c̃_t but then squashing c_t itself — tanh is applied only when producing h_t, and c_t is unbounded.
• Counting LSTM parameters as n(m+n+1) instead of 4n(m+n+1), or forgetting to double for bidirectional.
• Claiming an LSTM "removes" the vanishing-gradient problem — it greatly mitigates it; gradients through the gates and W paths still decay, and very long dependencies can still be missed.
• Computing the loss over padded time steps, or feeding padding into the state — inflating or wrecking metrics; always mask or pack.
• Using h_T of a bidirectional model as "the summary" without noticing the backward direction's summary lives at the *first* time step — concatenate both, or pool over all steps.
• Scaling features before the train/test split (fitting the scaler on all the data), or shuffling time-series rows so future data leaks into training — the most common forecasting error in labs and interviews.
• Reporting perplexity from a loss measured in bits as e^loss, or quoting perplexity across models with different vocabularies or tokenisers as if comparable.

Related topics

Neural networks and backpropagation · Transformers and attention · Training deep networks: optimisers, normalisation and regularisation · Time-series forecasting and ARIMA · Natural language processing and word embeddings · Speech recognition and CTC