Stochastic processes and Markov chains

In short

A stochastic process is a family of random variables {X(t) : t ∈ T} indexed by time, describing how a random quantity evolves; a Markov chain is a stochastic process whose future depends only on the present state and not on the path that led there. For actuaries this is the engine behind no-claim discount (NCD) systems in motor insurance, multiple-state health and sickness models, pension-scheme member movements, credit-rating migration and claim-count modelling by the Poisson process. It is examined in IFoA/IAI Subject CS2 (Risk Modelling and Survival Analysis), in B.Sc Actuarial Science probability papers, and in GATE Statistics (ST) and ISI/CMI entrance papers. Employers in India testing it include ICICI Lombard, HDFC ERGO, Bajaj Allianz, LIC, SBI Life, Swiss Re, Munich Re, WTW, Mercer, Aon, Milliman and the actuarial arms of EY, Deloitte and KPMG. The topic is pure probability modelling — distributions and estimation are covered elsewhere; here the focus is dynamics over time and the long-run behaviour that follows.

Explained

Stochastic processes: definitions and classification

• A stochastic process is a collection {X(t), t ∈ T} of random variables on a common probability space; state space S is the set of values X(t) can take, time set T the index set.
• Classification by both axes gives four types: discrete time–discrete state (a Markov chain, e.g. NCD level each year), discrete time–continuous state (annual fund value), continuous time–discrete state (number of claims at time t, a Markov jump process), continuous time–continuous state (Brownian motion of a share price).
• A sample path (realisation) is one observed trajectory; a process is characterised by the joint distribution of (X(t₁), …, X(tₙ)) for all finite sets of times.
• Stationarity (strict): the joint distribution of (X(t₁+h), …, X(tₙ+h)) is independent of h. Weak (second-order) stationarity: E[X(t)] constant and Cov(X(t), X(t+h)) a function of h only.
• Increments: independent increments means X(t₂) − X(t₁), X(t₄) − X(t₃) are independent for disjoint intervals; stationary increments means the distribution of X(t+h) − X(t) depends only on h.
• Counting process N(t): non-decreasing, integer valued, N(0) = 0 — counts events (claims, deaths, arrivals) up to time t.
• Filtration/history Fₜ (or Hₜ): all information up to time t. A martingale satisfies E[X(t+s) | Fₜ] = X(t) — a fair game; examined as a property, not developed in depth at undergraduate level.

The Markov property and discrete-time Markov chains

• Markov property: P(Xₙ₊₁ = j | Xₙ = i, Xₙ₋₁ = iₙ₋₁, …, X₀ = i₀) = P(Xₙ₊₁ = j | Xₙ = i). The present screens off the past — "memoryless" in state, not necessarily in time spent.
• One-step transition probability pᵢⱼ(n) = P(Xₙ₊₁ = j | Xₙ = i). The chain is time-homogeneous if pᵢⱼ(n) = pᵢⱼ for all n.
• Transition probability matrix (TPM) P = [pᵢⱼ] is stochastic: pᵢⱼ ≥ 0 and every row sums to 1. A doubly stochastic matrix also has column sums 1.
• The chain is fully specified by P and the initial distribution π⁽⁰⁾ (a row vector of P(X₀ = i)).
• Transition graph: states as nodes, arrows labelled with pᵢⱼ > 0 — the fastest way to read off irreducibility, absorbing states and periodicity in an exam.
• Augmenting the state space: a non-Markov process can often be made Markov by enlarging the state. Example: an NCD system where the discount depends on the last *two* years' claims is not Markov in "discount level" but is Markov in the pair (level, claim-free last year?). This trick is a standard CS2 exam question.
• Markov chain vs Markov jump process: the chain moves at fixed integer times; the jump process moves in continuous time with exponential holding times. A Markov jump chain is the embedded sequence of states visited, ignoring holding times.

Chapman–Kolmogorov, n-step probabilities and classification of states

• n-step probability pᵢⱼ⁽ⁿ⁾ = P(Xₘ₊ₙ = j | Xₘ = i); in matrix form P⁽ⁿ⁾ = Pⁿ.
• Chapman–Kolmogorov equations: pᵢⱼ⁽ᵐ⁺ⁿ⁾ = Σₖ pᵢₖ⁽ᵐ⁾ pₖⱼ⁽ⁿ⁾ — condition on the state at the intermediate time; the matrix statement is P^m+n = Pᵐ Pⁿ.
• Unconditional distribution: π⁽ⁿ⁾ = π⁽⁰⁾ Pⁿ (row-vector convention; some textbooks use column vectors and Pᵀ — state your convention).
• Accessibility: j is accessible from i (i → j) if pᵢⱼ⁽ⁿ⁾ > 0 for some n ≥ 0. States communicate (i ↔ j) if each is accessible from the other; communication is an equivalence relation and partitions S into communicating classes.
• Irreducible chain: one single class — every state reachable from every other.
• A class is closed if no state outside it is accessible from inside; a single-state closed class is an absorbing state (pᵢᵢ = 1).
• Recurrent (persistent) state: P(return to i eventually) = 1. Transient: that probability < 1, so i is visited finitely often. Criterion: i is recurrent iff Σₙ pᵢᵢ⁽ⁿ⁾ = ∞.
• Positive recurrent if the mean recurrence time μᵢ = E[time to return] is finite; null recurrent if it is infinite. A finite irreducible chain is always positive recurrent.
• Period d(i) = gcd{n ≥ 1 : pᵢᵢ⁽ⁿ⁾ > 0}; aperiodic if d = 1. Periodicity is a class property, as are recurrence and transience.
• Ergodic chain: irreducible, aperiodic and positive recurrent — the case in which a unique limiting distribution exists.
• Simple random walk on the integers with up-probability p: irreducible, period 2, recurrent iff p = ½ (and then null recurrent).

Stationary and limiting distributions

• A stationary (invariant) distribution π is a probability row vector with π = πP, i.e. πⱼ = Σᵢ πᵢ pᵢⱼ, together with Σⱼ πⱼ = 1. Once the chain is in π it stays in π.
• Solving: write the |S| balance equations, drop one (they are linearly dependent since rows sum to 1) and replace it with the normalisation Σπⱼ = 1.
• A limiting distribution is limₙ→∞ pᵢⱼ⁽ⁿ⁾ = πⱼ, independent of the starting state i. For a finite ergodic chain the limiting distribution exists, equals the unique stationary distribution, and Pⁿ converges to a matrix with every row equal to π.
• If the chain is irreducible and positive recurrent but periodic, π still exists and is unique, but Pⁿ does not converge — π is then only the long-run *average* proportion of time in each state.
• Mean recurrence time μⱼ = 1/πⱼ for an irreducible positive recurrent chain — a one-line result that answers "how often, on average, does a motorist fall back to 0% discount?"
• A reducible chain may have many stationary distributions (one per closed class, and mixtures of them); the long-run answer then depends on the starting state through absorption probabilities.
• Reversibility / detailed balance: if πᵢpᵢⱼ = πⱼpⱼᵢ for all i, j then π is stationary; a useful shortcut for birth–death type chains and for random walks on graphs.

Absorption, first passage and the gambler's ruin

• First passage time Tᵢⱼ = min{n ≥ 1 : Xₙ = j | X₀ = i}; first passage probability fᵢⱼ⁽ⁿ⁾ = P(Tᵢⱼ = n). Recurrence of i means Σₙ fᵢᵢ⁽ⁿ⁾ = 1.
• Absorption probability from transient state i into absorbing state a: condition on the first step — aᵢ = pᵢₐ + Σ_{j transient} pᵢⱼ aⱼ, solved as a linear system with aₐ = 1 and 0 at other absorbing states.
• Expected time to absorption from i: mᵢ = 1 + Σ_{j transient} pᵢⱼ mⱼ, with mᵢ = 0 for absorbing states.
• Canonical form: order states as transient then absorbing, P = [[Q, R],[0, I]]. The fundamental matrix M = (I − Q)⁻¹ gives expected visits; row sums of M are expected times to absorption, and MR gives absorption probabilities by state. A clean matrix answer where hand-solving equations is slow.
• Gambler's ruin: capital j out of total a, win probability p each round, q = 1 − p. Ruin probability and expected duration have closed forms (see formula boxes) — the archetype of an absorbing chain and the ancestor of ruin theory in general insurance.
• Actuarial use: absorption = death, lapse, full recovery or claim settlement in a multiple-state model; expected time to absorption = expected duration of a sickness claim.

Continuous-time Markov chains, the Poisson process and birth–death processes

• A Markov jump process on a discrete state space has the Markov property in continuous time; the time spent in state i (holding time) is Exponential with rate λᵢ, and the exponential's memorylessness is what makes the Markov property possible.
• Transition rates (generator) matrix Q = [qᵢⱼ]: for i ≠ j, qᵢⱼ ≥ 0 is the instantaneous rate i → j; qᵢᵢ = −Σ_{j≠i} qᵢⱼ, so every row of Q sums to zero.
• Kolmogorov forward equations: dP(t)/dt = P(t)Q; backward equations: dP(t)/dt = QP(t), with P(0) = I. Formally P(t) = e^{Qt}.
• Stationary distribution of a jump process solves πQ = 0 with Σπᵢ = 1 (contrast π = πP in discrete time — a very common exam slip).
• Poisson process with rate λ: a counting process with N(0) = 0, independent and stationary increments, N(t) ~ Poisson(λt), and inter-arrival times i.i.d. Exponential(λ). It is a pure birth process with qᵢ,ᵢ₊₁ = λ.
• Properties: superposition of independent Poisson processes with rates λ₁, λ₂ is Poisson(λ₁+λ₂); thinning each event independently with probability p gives Poisson(pλ); given N(t) = n, the n event times are distributed as n i.i.d. Uniform(0, t) order statistics.
• Non-homogeneous Poisson process: rate λ(t), so N(t) ~ Poisson(∫₀ᵗ λ(u)du) — used for seasonal claim arrivals (monsoon motor claims in India).
• Compound Poisson process S(t) = Σ_{i=1}^{N(t)} Xᵢ: aggregate claim amount, the link from this topic into risk theory and ruin probability.
• Birth–death process: moves only to adjacent states, birth rate λᵢ, death rate μᵢ. Detailed balance gives πᵢ₊₁ = πᵢ λᵢ/μᵢ₊₁. The M/M/1 queue (λᵢ = λ, μᵢ = μ, ρ = λ/μ < 1) has πₙ = (1 − ρ)ρⁿ and mean number in the system L = ρ/(1 − ρ) (mean number waiting Lq = ρ²/(1 − ρ)).
• Time-inhomogeneous models (rates depending on age) underlie the multiple-state survival models used for Indian mortality tables (IALM — Indian Assured Lives Mortality, published by the IAI).

Estimation, simulation and model limitations

• Estimating a TPM from data: p̂ᵢⱼ = nᵢⱼ / nᵢ where nᵢⱼ is the count of observed i → j transitions and nᵢ = Σⱼ nᵢⱼ — the MLE for a time-homogeneous chain. Rows with few observations give unstable estimates.
• For a jump process observed continuously, λ̂ᵢⱼ = (transitions i → j) / (total waiting time in i) — the occupancy-time MLE.
• Simulation (discrete time): draw U ~ Uniform(0,1) and use the inverse-CDF of the current row of P. Continuous time: draw holding time ~ Exp(−qᵢᵢ) then pick the next state with probability qᵢⱼ/(−qᵢᵢ).
• Advantages: few parameters, tractable algebra, closed-form long-run answers, natural fit to "state of a policy/life" problems, easily extended to many states.
• Limitations: the Markov assumption is often false (duration dependence — a motorist who has been claim-free for five years differs from one claim-free for one); time-homogeneity may fail (inflation, regulation); exponential holding times are restrictive; large state spaces need lots of data; semi-Markov models (holding-time distribution depending on the state) relax this.
• Indian practice: the IRDAI's motor own-damage NCD grid (20/25/35/45/50% after 1 to 5 consecutive claim-free years) is the classic Indian Markov chain; third-party premiums are notified by the Ministry of Road Transport and Highways in consultation with IRDAI and are not discounted. IRDAI Actuarial Report and Abstract requirements, and the IAI's Actuarial Practice Standards, govern the use of such multiple-state models in valuation; detailed rules change by circular, so check the current IRDAI circular for exact grids.

Key points

• Stochastic process - an indexed family of random variables; classified by discrete/continuous time and discrete/continuous state space.
• Markov property - the conditional distribution of the future given the present is free of the past.
• TPM - square non-negative matrix with unit row sums; with the initial distribution it determines the whole chain.
• Chapman–Kolmogorov - P^m+n = PᵐPⁿ; gives n-step probabilities by matrix powers.
• Distribution at time n - π⁽ⁿ⁾ = π⁽⁰⁾Pⁿ, a row vector times a matrix power.
• Communicating class - equivalence class under mutual accessibility; recurrence, transience and period are class properties.
• Irreducible - a single communicating class; aperiodic - period 1; ergodic - irreducible, aperiodic, positive recurrent.
• Recurrent vs transient - return with probability 1 (Σpᵢᵢ⁽ⁿ⁾ = ∞) versus < 1; finite irreducible chains are positive recurrent.
• Absorbing state - pᵢᵢ = 1; once entered, never left (death, lapse, ruin).
• Stationary distribution - π = πP with Σπ = 1; unique for an irreducible positive recurrent chain.
• Limiting distribution - exists and equals π only if the chain is also aperiodic; periodic chains have π as a time-average only.
• Mean recurrence time - μⱼ = 1/πⱼ for an irreducible positive recurrent chain.
• Fundamental matrix - M = (I − Q)⁻¹ in canonical form; row sums give expected times to absorption, MR gives absorption probabilities.
• Generator matrix Q - rows sum to zero; stationary distribution solves πQ = 0, not π = πP.
• Kolmogorov equations - forward P′(t) = P(t)Q, backward P′(t) = QP(t), P(0) = I.
• Poisson process - rate λ, N(t) ~ Poisson(λt), Exponential(λ) inter-arrival times; closed under superposition and thinning.
• Birth–death process - adjacent moves only; πᵢ₊₁ = πᵢλᵢ/μᵢ₊₁ by detailed balance; M/M/1 gives πₙ = (1 − ρ)ρⁿ.
• MLE of transitions - p̂ᵢⱼ = nᵢⱼ/nᵢ in discrete time, λ̂ᵢⱼ = transitions/waiting time in continuous time.

Formulas and examples

Example 1 — Two-state weather/lapse chain, n-step and stationary distribution. A policy is either "in force" (state 1) or "lapsed-but-revivable" (state 2). P = [[0.9, 0.1],[0.4, 0.6]], so a = 0.1, b = 0.4.

• P² = [[0.9×0.9 + 0.1×0.4, 0.9×0.1 + 0.1×0.6], [0.4×0.9 + 0.6×0.4, 0.4×0.1 + 0.6×0.6]] = [[0.85, 0.15],[0.60, 0.40]].
• So a policy in force now is in force in 2 years with probability 0.85.
• Stationary: π₁ = b/(a+b) = 0.4/0.5 = 0.8, π₂ = 0.1/0.5 = 0.2. Check: 0.8×0.9 + 0.2×0.4 = 0.72 + 0.08 = 0.80 ✓.
• Mean recurrence time of the lapsed state μ₂ = 1/0.2 = 5 years.
• Starting distribution π⁽⁰⁾ = (0.6, 0.4): π⁽¹⁾ = (0.6×0.9 + 0.4×0.4, 0.6×0.1 + 0.4×0.6) = (0.70, 0.30).

Example 2 — Motor NCD chain (three discount levels), long-run premium cost. Levels 0%, 25%, 50%. A claim-free year moves up one level (50% stays); a year with a claim drops to 0%. Claim probability 0.2 each year, independent. P (rows/cols in order 0, 25, 50) = [[0.2, 0.8, 0],[0.2, 0, 0.8],[0.2, 0, 0.8]].

• Balance equations: π₀ = 0.2(π₀+π₁+π₂) = 0.2 → π₀ = 0.2.
• π₂₅ = 0.8π₀ = 0.8 × 0.2 = 0.16; π₅₀ = 0.8π₂₅ + 0.8π₅₀ ⇒ 0.2π₅₀ = 0.8 × 0.16 = 0.128 ⇒ π₅₀ = 0.64.
• Check sum: 0.2 + 0.16 + 0.64 = 1.00 ✓.
• Long-run average discount = 0.2×0 + 0.16×0.25 + 0.64×0.50 = 0.04 + 0.32 = 0.36, i.e. 36%.
• On a full own-damage premium of ₹12,000, the expected steady-state premium is 12,000 × 0.64 = ₹7,680 per year.
• Mean recurrence time of level 0 = 1/0.2 = 5 years — a motorist returns to nil discount once every five years on average.

Example 3 — Absorption: expected duration of a sickness claim. States: H = healthy, S = sick, D = dead (absorbing), with annual transitions P = [[0.9, 0.08, 0.02],[0.3, 0.6, 0.1],[0, 0, 1]].

• Transient block Q = [[0.9, 0.08],[0.3, 0.6]]. I − Q = [[0.1, −0.08],[−0.3, 0.4]], det = 0.1×0.4 − (−0.08)(−0.3) = 0.04 − 0.024 = 0.016.
• M = (I − Q)⁻¹ = (1/0.016)[[0.4, 0.08],[0.3, 0.1]] = [[25, 5],[18.75, 6.25]].
• Expected years until death starting healthy = 25 + 5 = 30 years; starting sick = 18.75 + 6.25 = 25 years.
• Expected years spent sick before death, starting healthy = M[H][S] = 5 years. At a sickness benefit of ₹2,00,000 per year this is an expected ₹10,00,000 of benefit (before discounting).

Example 4 — Gambler's ruin for a small insurer's surplus. Surplus 4 units, ruin at 0, target 10 units; each period the surplus moves ±1 with p = 0.5.

• p = q, so ruin probability = 1 − j/a = 1 − 4/10 = 0.6.
• Expected number of periods until either barrier = j(a − j) = 4 × 6 = 24.
• Now take p = 0.6 (premium loading), q = 0.4, q/p = 2/3: ruin prob = [(2/3)⁴ − (2/3)¹⁰]/[1 − (2/3)¹⁰] = (0.197531 − 0.017342)/(1 − 0.017342) = 0.180189/0.982658 = 0.1834.
• A 10-percentage-point edge cuts the ruin probability from 60% to about 18%.

Example 5 — Poisson claims and a two-state jump process. Motor claims arrive as a Poisson process with λ = 3 per day at a Pune service centre.

• P(no claim in a day) = e^{−3} = 0.0498; P(exactly 2 in a day) = e^{−3}3²/2! = 0.0498 × 4.5 = 0.2240.
• E[N(7)] = 21 claims a week, Var = 21, sd = √21 ≈ 4.58.
• Thinning: if 30% are total-loss claims, those form a Poisson process with rate 0.9 per day.
• Jump process, states Active (A) and Disabled (B), rates q_AB = 0.05, q_BA = 0.20 per year. Q = [[−0.05, 0.05],[0.20, −0.20]].
• πQ = 0: −0.05π_A + 0.20π_B = 0 ⇒ π_A = 4π_B; with π_A + π_B = 1, π_B = 0.2, π_A = 0.8.
• Mean holding time in A = 1/0.05 = 20 years; in B = 1/0.20 = 5 years.

Where it is used

• General insurance pricing (motor): the IRDAI no-claim-discount grid modelled as a Markov chain to find the steady-state discount, premium leakage and the correct base rate — core work at ICICI Lombard, HDFC ERGO, Bajaj Allianz, Go Digit and TATA AIG.
• Life and health insurance: multiple-state models (healthy–sick–dead, active–disabled–lapsed) for critical illness, disability income and long-term care pricing and reserving, and for lapse/persistency studies reported under IRDAI returns.
• Pensions and employee benefits: member movement between active, deferred, pensioner and withdrawn states for valuations under AS 15 / Ind AS 19 — WTW, Mercer, Aon, K A Pandit, Milliman.
• Banking and credit risk: rating-migration matrices (CRISIL, ICRA, CARE ratings) feeding expected credit loss under Ind AS 109; Markov chains for NPA bucket transitions (standard → SMA-1 → SMA-2 → NPA) in RBI reporting.
• Operations, telecom and IT: M/M/1 and M/M/c queueing for call-centre staffing and server capacity; Google's PageRank is a stationary distribution; Markov chains underlie MCMC in Bayesian analytics roles.
• Reinsurance and capital modelling: compound Poisson aggregate loss models and ruin probability in IRDAI solvency and internal capital work at Swiss Re, Munich Re and GIC Re.

Interview questions

• Q: State the Markov property in one sentence. A: Given the present state, the conditional distribution of the future is independent of the past — the path by which the state was reached carries no extra information.
• Q: Why must every row of a TPM sum to one but every row of a generator matrix sum to zero? A: A TPM row is a probability distribution over the next state; a generator row holds rates, with the diagonal set to minus the total exit rate so probabilities stay normalised as t increases.
• Q: Difference between a stationary and a limiting distribution? A: A stationary distribution satisfies π = πP and always exists for a finite chain; a limiting distribution is the limit of Pⁿ and exists only if the chain is also aperiodic (and irreducible for it to be unique and start-independent).
• Q: An NCD system where the discount depends on the claims record of the last two years — is it Markov? A: Not in "discount level" alone, but it becomes Markov if the state is augmented to include whether the previous year was claim-free.
• Q: Why are holding times in a continuous-time Markov chain exponential? A: Only the exponential distribution is memoryless, which is exactly what the Markov property requires in continuous time.
• Q: A chain has two absorbing states. Does it have a unique stationary distribution? A: No — each absorbing state gives a stationary distribution and so does any mixture, so the long-run outcome depends on the starting state through the absorption probabilities.
• Q: How would you estimate a transition matrix from policy data? A: Count transitions and divide by the row total, p̂ᵢⱼ = nᵢⱼ/nᵢ, the MLE under time-homogeneity; check the counts are large enough and test homogeneity across calendar periods.
• Q: What does the fundamental matrix give you? A: With P = [[Q,R],[0,I]], M = (I − Q)⁻¹ gives expected numbers of visits to each transient state; row sums are expected times to absorption and MR gives absorption probabilities.
• Q: In a simple random walk with p = ½, is state 0 recurrent? A: Yes, but null recurrent — the walk returns with probability 1 yet the expected return time is infinite, so no stationary distribution exists.
• Q: Superposition and thinning of a Poisson process? A: Independent Poisson processes with rates λ₁ and λ₂ superpose to a Poisson process of rate λ₁+λ₂; retaining each event independently with probability p thins it to rate pλ.
• Q: Where does the Markov assumption break down in insurance? A: Wherever duration matters — recovery rates depend on how long someone has been sick, and lapse rates on policy duration; semi-Markov models or extra state variables are needed.
• Q: How do you simulate a Markov jump process? A: From state i, draw a holding time from Exponential(−qᵢᵢ), then jump to state j with probability qᵢⱼ/(−qᵢᵢ), and repeat.

Common mistakes

• Writing πP = π as a column system (Pπ = π) or forgetting the normalising condition Σπ = 1, giving the zero or a scaled solution.
• Solving π = πP for a continuous-time process instead of πQ = 0, or forgetting that Q rows sum to zero.
• Claiming Pⁿ converges for a periodic chain; period 2 chains oscillate forever even though π exists.
• Multiplying matrices in the wrong order or mis-indexing pᵢⱼ as "from j to i" — fix the row = from, column = to convention before starting.
• Assuming an irreducible chain whenever a stationary distribution is asked for: check for absorbing states and closed classes first, since reducible chains have multiple solutions.
• Treating a process as Markov because it "looks" memoryless — test whether duration or history affects the next transition, and augment the state space if it does.
• Confusing the number of transitions with elapsed time in the continuous case, and using p̂ᵢⱼ = nᵢⱼ/nᵢ where the occupancy-time estimator λ̂ᵢⱼ = nᵢⱼ/(waiting time in i) is required.

Related topics

Probability theory and Bayes' theorem · Random variables and expectation · Probability distributions · Time series analysis · Survival models and life contingencies · Risk theory and ruin probability