Caching, queues and CDNs

In short

Caching keeps a copy of expensive data close to where it is needed, queues decouple a fast producer from a slower consumer, and a CDN pushes static content to servers near the user. Together they are the three cheapest ways to make a web system fast and resilient without buying a bigger database. Universities examine this under web technologies, cloud computing and distributed systems; placement interviews at Flipkart, Zomato, Swiggy, Paytm, PhonePe, Razorpay, Zoho, Freshworks, TCS Digital and Infosys Power Programmer ask it in almost every system-design round. Indian traffic patterns make it concrete: IRCTC Tatkal at 10:00, a BookMyShow ticket drop, or an IPL stream on JioHotstar all survive on cache hit ratio and queue depth, not on raw CPU.

Explained

What a cache is and the layers it lives in

• A cache is a small, fast store holding a subset of data from a slower store, keyed for quick lookup. A hit is served from cache; a miss goes to the origin and usually populates the cache.
• Locality is why it works: *temporal* (recently used data is used again) and *spatial* (nearby data is used together).
• Layers, nearest to the user first:
  1. Browser cache - HTTP responses on the user's disk, governed by Cache-Control.
  2. CDN / edge cache - shared, in a nearby city (Mumbai, Chennai, Delhi, Bengaluru edges).
  3. Reverse proxy cache - Nginx, Varnish, in front of the app.
  4. Application / in-process cache - a HashMap or Guava/lru-cache inside one server process; fastest, but not shared and lost on restart.
  5. Distributed cache - Redis or Memcached, shared by all app servers, typically 0.2-1 ms away.
  6. Database cache - buffer pool, query cache, materialised views.
• Private vs shared: a browser cache is private to one user (safe for personalised data); a CDN or Redis entry is shared (never put another user's personal data in a shared cache key).

Cache read and write strategies

• Cache-aside (lazy loading) - the application checks the cache; on a miss it reads the database and writes the value back. Most common; only requested data is cached; a miss costs two trips.
• Read-through - the cache library itself fetches from the database on a miss; application code sees a single call.
• Write-through - every write goes to cache and database synchronously. Cache never stale; writes slower.
• Write-behind (write-back) - write to cache, flush to the database asynchronously. Fastest writes, highest risk of data loss if the cache node dies.
• Write-around - write only to the database and let the next read populate the cache; good for write-heavy data that is rarely read.
• Refresh-ahead - the cache proactively refreshes a hot entry just before its TTL expires, hiding the miss latency.

Eviction, expiry and invalidation

• A cache is bounded, so entries must leave. Eviction policies:
  • LRU - evict the least recently used; the usual choice in Redis (allkeys-lru - note Redis's own default is noeviction, which rejects writes once maxmemory is reached), the default in Memcached, and the one asked in exams.
  • LFU - least frequently used; better when a few keys are permanently hot (Redis allkeys-lfu uses an approximated counter).
  • FIFO - oldest inserted first; simple, ignores usage.
  • Random - surprisingly decent and O(1); used as allkeys-random.
  • TTL-based - volatile-lru / volatile-ttl evict only keys that carry an expiry.
• TTL (time to live) is the main correctness knob: short TTL = fresher but more misses; long TTL = faster but staler. Pick TTL from how stale the business can tolerate the data (a product price 30 s, a city list 24 h).
• Invalidation is the hard part. Three options: (a) TTL expiry only, (b) explicit delete on write (delete the key, let the next read repopulate - safer than updating it), (c) versioned keys / cache busting - embed a version or content hash in the key or filename (app.9f2c1a.js) so a new deploy simply uses new keys.
• Belady's optimal (OPT) replacement - evict the entry needed furthest in the future; unimplementable (needs the future) but the theoretical upper bound against which LRU is measured in GATE questions.
• Stampede problems and fixes:
  • Thundering herd / cache stampede - a hot key expires and thousands of requests hit the database at once. Fix: a mutex/lock so one request repopulates, stale-while-revalidate (serve the old value while refreshing), or jittered TTLs.
  • Cache penetration - repeated requests for a key that does not exist anywhere. Fix: cache the negative result briefly, or a Bloom filter in front.
  • Cache avalanche - many keys expiring at the same instant; fix with randomised TTLs.
• Consistency: a cache makes the system eventually consistent. Cache-aside has a known race (reader caches an old value between a writer's DB update and key delete); mitigated by delete-after-commit, short TTL, or write-through for critical data.

HTTP caching and CDNs

• A CDN is a geographically distributed network of PoPs (points of presence) holding cached copies of your content. The user's DNS or anycast route sends them to the nearest edge, cutting RTT - the dominant cost for a page of 60 small files.
• Pull CDN (origin pull) - the edge fetches from your origin on the first miss and caches it; the default for websites. Push CDN - you upload content to the CDN; used for large videos and software downloads.
• Key HTTP headers:
  • Cache-Control: max-age=31536000, immutable for hashed static assets; no-store for personal data; private for per-user responses; s-maxage targets shared caches only.
  • ETag + If-None-Match, or Last-Modified + If-Modified-Since → a 304 Not Modified revalidation that saves the body but not the round trip.
  • stale-while-revalidate and stale-if-error let an edge serve a slightly old copy instead of a slow page or a 503.
  • Vary: Accept-Encoding (and carefully, Vary: Accept-Language) tells shared caches which request headers change the response; a Vary on a high-cardinality header destroys hit ratio.
• Cache key at the edge = host + path + chosen query parameters (+ Vary headers). Stripping tracking parameters such as utm_* from the cache key raises hit ratio.
• Purge / invalidation at a CDN: purge by URL, by tag/surrogate key, or purge everything (slow and expensive - avoid by versioning filenames).
• Dynamic acceleration: even uncacheable HTML gains from a CDN through TLS termination at the edge, keep-alive connections and optimised backbone routing; edge compute (Cloudflare Workers, Lambda@Edge, CloudFront Functions) runs small logic at the PoP.
• Also at the edge: TLS, HTTP/2 and HTTP/3 (QUIC), Brotli/gzip compression, image resizing, WAF and DDoS absorption. Indian context: Cloudflare, Akamai, AWS CloudFront, Google Cloud CDN and Indian providers such as Jio CDN and Tata Communications all run PoPs in Mumbai, Chennai, Delhi NCR, Bengaluru and Hyderabad.

Message queues and asynchronous processing

• A message queue holds messages between a producer and a consumer so neither waits for the other. It gives decoupling (producer does not know the consumer), buffering (absorbs a spike), load levelling, retries and independent scaling.
• Synchronous vs asynchronous: sending a confirmation SMS inside the checkout request makes the user wait for the SMS gateway; pushing a job to a queue returns in milliseconds and the worker sends it a moment later.
• Messaging models:
  • Point-to-point (work queue) - each message is consumed by exactly one of N competing consumers; used for task distribution.
  • Publish-subscribe (fan-out) - every subscriber gets a copy; one "order placed" event drives invoicing, notification and analytics independently.
  • Request-reply over a queue, using a correlation id and a reply queue.
• Broker styles:
  • Classic broker (RabbitMQ, ActiveMQ) - AMQP exchanges route to queues; the broker deletes a message after acknowledgement. Rich routing (direct, topic, fanout, headers), per-message TTL, priorities.
  • Log-based (Apache Kafka, AWS Kinesis) - an append-only, partitioned commit log; consumers track an offset and can replay history. Built for very high throughput and multiple independent consumer groups.
  • Managed cloud queues - AWS SQS (standard = at-least-once, possibly out of order; FIFO = ordered, deduplicated), Google Pub/Sub, Azure Service Bus, AWS SNS for fan-out.
• Delivery semantics: at-most-once (may lose), at-least-once (may duplicate - the practical default), exactly-once (hard and costly; usually faked as at-least-once + idempotent consumers).
• Idempotency is the exam answer to duplicates: give every message a unique id (or use the order id as a key), record processed ids, and make the handler safe to run twice. A payment capture must never double-charge on a redelivery.
• Acknowledgement and failure: consumer acks after successful processing; an unacked message becomes visible again after a visibility timeout (SQS) and is redelivered. After N failures the message goes to a dead letter queue (DLQ) for inspection instead of blocking the queue (poison message).
• Ordering: global order is expensive. Kafka guarantees order *within a partition*, so partition by a key (user id, order id) to keep each entity's events ordered while still running in parallel.
• Backpressure and queue depth: if arrival rate exceeds service rate, the queue grows without bound and latency grows with it. Responses: autoscale consumers, shed or throttle load, prioritise queues, or reject with HTTP 429.
• Other patterns: delay/scheduled queues (retry a failed webhook after 5 min), exponential backoff with jitter, outbox pattern (write the event to a DB table in the same transaction, then publish - avoids "DB committed but message lost"), saga/choreography for multi-service workflows, event sourcing and CQRS built on a log.
• Limitations: eventual consistency, harder debugging and tracing, duplicate handling, message-size limits (SQS 256 KB - put large payloads in S3 and send the pointer), operational cost of running a broker.

Putting the three together, and what to measure

• Typical request path: browser cache → CDN edge → load balancer → app server → Redis → database; writes go to the database and drop an event on a queue for everything that need not be instant.
• What is cacheable: read-heavy, slow-to-compute, tolerant of slight staleness, and shared by many users. What is not: per-user secrets in a shared cache, data requiring strict read-your-writes, or a key written far more often than read.
• Metrics: hit ratio (per layer), p50/p95/p99 latency, origin offload %, eviction rate, Redis memory and keyspace_misses, queue depth and consumer lag, oldest-message age, DLQ count, egress GB.
• Cost logic in India: CDN egress is usually billed per GB and is far cheaper than origin bandwidth plus compute; a managed Redis (ElastiCache / Azure Cache) node for small workloads runs a few thousand rupees a month, while the database it protects costs many times more - so caching is normally the cheapest performance rupee. Check current provider price lists, since rates change.

Key points

• Cache - a fast copy of slow data; measured by hit ratio = hits/(hits+misses).
• Cache-aside - app reads cache, on miss reads DB and backfills; the default pattern.
• Write-through vs write-back - synchronous (safe, slower) vs asynchronous (fast, risk of loss).
• LRU - evict least recently used; LFU for permanently hot keys; OPT/Belady is the unachievable ideal.
• TTL - expiry time; the trade-off dial between freshness and hit ratio.
• Invalidation - prefer deleting the key on write and versioned asset filenames over trying to update cached copies.
• Thundering herd - a hot key expiring floods the origin; fix with a lock, jittered TTL or stale-while-revalidate.
• Cache penetration - misses for keys that do not exist; cache the negative or use a Bloom filter.
• CDN - edge PoPs serving content near the user; pull (origin-pull) for sites, push for large media.
• Cache-Control / ETag / 304 - the HTTP controls for max-age, revalidation and shared-cache behaviour (s-maxage, private, no-store).
• Vary - declares which request headers change the response; a high-cardinality Vary kills hit ratio.
• Queue - buffer between producer and consumer; gives decoupling, load levelling and retries.
• Point-to-point vs pub-sub - one consumer per message vs a copy to every subscriber.
• Kafka vs RabbitMQ - replayable partitioned log with offsets vs a routing broker that deletes on ack.
• At-least-once + idempotent consumer - the practical substitute for exactly-once delivery.
• DLQ - where a repeatedly failing (poison) message goes so the queue keeps moving.
• Consumer lag / queue depth - the health signal; growing lag means arrival rate exceeds service rate.
• Little's Law - queue length L = arrival rate λ × wait time W; the one formula to quote about queues.

Formulas and examples

Example 1 - Effective latency from a cache. A product API serves 5,000 requests/s. Redis responds in 1 ms, the database in 40 ms. Hit ratio is 92%.

• T_eff = 0.92 × 1 + 0.08 × 40 = 0.92 + 3.2 = 4.12 ms.
• Origin (DB) load = 0.08 × 5,000 = 400 queries/s instead of 5,000 - that is (5,000 − 400)/5,000 = a 92% reduction, exactly equal to the hit ratio.
• Raise the hit ratio to 97%: T_eff = 0.97 × 1 + 0.03 × 40 = 0.97 + 1.2 = 2.17 ms, DB load = 150 q/s. The last 5 points of hit ratio cut database load by 62.5% (from 400 to 150) - which is why hit ratio work pays off most at the top end.

Example 2 - LRU trace (exam style). Cache of size 3, access sequence: A B C A D B E. Count hits and misses with LRU.

• Hits = 1, misses = 6 → hit ratio = 1/7 ≈ 14.3%. With OPT (Belady) the same trace gives 2 hits, showing LRU is not optimal.

Example 3 - Sizing consumers with Little's Law. A Diwali sale produces order-confirmation emails at λ = 600 messages/s. One worker handles μ = 25 messages/s.

• Minimum consumers c ≥ 600/25 = 24. Running exactly 24 gives ρ = 1 - unstable, so run 30 workers → ρ = 600/(30 × 25) = 0.8.
• Treating the 30 workers as one server of capacity 750/s, M/M/1 gives W = 1/(750 − 600) = 1/150 s ≈ 6.7 ms in system, and L = λW = 600 × 0.00667 = 4 messages in the system on average.
• If only 20 workers run, capacity = 500/s < 600/s: the backlog grows at 100 messages/s, i.e. 6,000 messages after one minute, and the oldest-message age climbs steadily - the classic "queue depth rising" alert.

Example 4 - CDN offload and cost. A news site serves 20 TB a month: 18 TB static (images, JS, CSS) and 2 TB HTML. The CDN caches static content at a 95% hit ratio; HTML is not cached.

• Bytes from the edge = 18 × 0.95 = 17.1 TB; from origin = 0.9 TB static + 2 TB HTML = 2.9 TB.
• Origin offload = 1 − 2.9/20 = 85.5%.
• At an illustrative ₹6/GB CDN rate, 17.1 TB ≈ 17,510 GB ≈ ₹1,05,060; the same bytes from origin bandwidth at, say, ₹10/GB would be ₹1,75,100 - plus the servers to produce them. Rates vary widely by provider and commitment; verify current pricing.

Example 5 - TTL and staleness budget. A cricket scorecard is cached for 10 s and fetched 12,000 times a minute.

• Origin fetches = 60/10 = 6 per minute per edge PoP → with 8 PoPs, 48 origin fetches/min against 12,000 user requests: hit ratio ≈ 1 − 48/12,000 = 99.6%.
• Worst-case staleness a user can see = TTL = 10 s (plus browser max-age, so keep browser max-age small, e.g. 5 s, and let s-maxage=10 apply at the edge).

// Cache-aside with a stampede lock and jittered TTL
async function getProduct(id) {
  const key = `product:v3:${id}`;              // v3 = version, cheap invalidation
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit);

  const lock = await redis.set(`${key}:lock`, '1', 'NX', 'EX', 10);
  if (!lock) {                                  // someone else is loading it
    await sleep(50);
    return getProduct(id);
  }
  const row = await db.findProduct(id);
  const ttl = 300 + Math.floor(Math.random() * 60);   // jitter avoids avalanche
  await redis.set(key, JSON.stringify(row), 'EX', ttl);
  await redis.del(`${key}:lock`);
  return row;
}

// Invalidate on write: delete, do not update
async function updateProduct(id, data) {
  await db.updateProduct(id, data);             // commit first
  await redis.del(`product:v3:${id}`);          // next read repopulates
}

// Idempotent consumer for an at-least-once queue
async function handle(msg) {
  const id = msg.messageId;
  const fresh = await db.insertIfAbsent('processed_messages', id);  // unique key
  if (!fresh) return ack(msg);                  // duplicate: already done
  try {
    await sendInvoice(msg.body);
    await ack(msg);
  } catch (e) {
    await nack(msg);                            // retried; after N tries → DLQ
  }
}

Where it is used

• E-commerce and food delivery (Flipkart, Myntra, Swiggy, Zomato): catalogue and menu pages in Redis and a CDN; order placement, notifications, invoicing and rider assignment pushed onto Kafka or SQS.
• Fintech and payments (Razorpay, PhonePe, Paytm, NPCI-linked flows): queues for webhook delivery with exponential backoff and DLQs, idempotency keys so a retried capture never double-charges; Redis for rate limiting and OTP throttling.
• Media and streaming (JioHotstar, SonyLIV, news portals): HLS/DASH video segments pushed to CDN PoPs in Mumbai, Chennai and Delhi; a live match is almost entirely edge-served.
• Ticketing and government portals (IRCTC, BookMyShow, UPSC/exam result sites): a static result page on a CDN turns a crushing load into edge traffic; booking requests enter a queue so the database sees a controlled rate.
• Job roles: backend developer, SRE/DevOps, cloud engineer, platform engineer - expected to configure Cache-Control, size an ElastiCache or MSK cluster, set alarms on consumer lag and write idempotent consumers.
• Compliance note: personal data cached or queued is covered by the Digital Personal Data Protection Act, 2023; store minimal personal data in shared caches, set short TTLs, and keep RBI's data-localisation requirement in mind for payment data - payment system data must be stored in India, which constrains where you cache or buffer it.

Interview questions

• Q: What is cache-aside and why is it the most common pattern? A: The application checks the cache, and on a miss reads the database and writes the value back. It caches only what is actually requested and keeps the cache out of the write path, so a cache failure degrades performance rather than correctness.
• Q: On a write, should you update the cached value or delete the key? A: Delete it. Updating risks writing a stale or racing value from a slower concurrent writer; deleting forces the next read to fetch the committed truth.
• Q: What is a cache stampede and how do you prevent it? A: Many requests miss simultaneously when a hot key expires and all hit the database. Prevent it with a mutex so only one request repopulates, stale-while-revalidate, refresh-ahead, or randomised (jittered) TTLs.
• Q: Difference between a CDN and a reverse proxy cache? A: Both cache in front of the origin, but a CDN is a globally distributed network of PoPs chosen by proximity to the user, while a reverse proxy sits in your own data centre and cuts origin load without cutting network distance.
• Q: How do you invalidate a JavaScript bundle on a CDN? A: Do not purge - change the filename using a content hash (app.9f2c1a.js) and serve it with a one-year immutable max-age; the HTML referencing it is cached for seconds.
• Q: What does Vary: Accept-Encoding do, and why is Vary: User-Agent dangerous? A: It tells shared caches to store a separate copy per value of that header. User-Agent has near-infinite values, so it fragments the cache and collapses the hit ratio.
• Q: Kafka or RabbitMQ for a payment event pipeline consumed by three teams? A: Kafka - the log retains events, each team runs its own consumer group with independent offsets and can replay history; RabbitMQ deletes a message once acknowledged.
• Q: Your system is at-least-once. How do you avoid charging a customer twice? A: Make the consumer idempotent: attach a unique idempotency key per payment, record it with a unique constraint at the start of processing, and skip the message if the key already exists.
• Q: What goes into a dead letter queue? A: A message that failed processing more than the configured number of times (a poison message), so it is moved aside for inspection instead of blocking or endlessly cycling through the queue.
• Q: Consumer lag keeps growing. What do you check? A: Whether arrival rate exceeds total consumer throughput - then add consumers (up to the partition count in Kafka), speed up the handler, or shed load; also check for a slow external dependency, a poison message or an unbalanced partition key.
• Q: Why can adding consumers beyond the partition count not help in Kafka? A: Within a consumer group each partition is assigned to exactly one consumer, so extra consumers stay idle; you must increase partitions to raise parallelism.
• Q: How would you pick a TTL? A: From the business tolerance for stale data, not from a feeling - take the maximum acceptable staleness, subtract any downstream browser cache, and add jitter so keys do not all expire together.
• Q: When should you *not* cache? A: When data is written far more often than read, when strict read-your-writes is required (balances, inventory decrements at checkout), or when the data is personal and the cache is shared.

Common mistakes

• Computing hit ratio over hits/misses but then quoting origin load as the hit ratio instead of (1 − H) × RPS.
• Caching per-user responses in a shared cache or CDN by forgetting Cache-Control: private / no-store - the classic bug that shows one user another's data.
• Setting a long max-age on HTML and then being unable to roll out a fix; the pattern is long TTL on hashed assets, seconds on HTML.
• Assuming "exactly-once delivery" exists and skipping idempotency, so a retry double-charges or sends a duplicate SMS.
• Giving every key the same TTL, producing a synchronised avalanche of misses on the hour.
• Acknowledging a message before the work succeeds (or using auto-ack), which silently loses messages when the consumer crashes.
• In Little's Law, mixing units - λ per minute with W in seconds - or applying M/M/1 formulas when ρ ≥ 1, where no steady-state answer exists.
• Treating a queue as a database: relying on a broker for long-term storage, or putting a 5 MB payload in a message instead of a pointer to object storage.

Related topics

System design basics for freshers · System design case studies · Cloud computing basics · AWS core services · REST APIs · Containers: Docker and Kubernetes