NoSQL databases

In short

NoSQL ("not only SQL") databases store data in models other than the relational table - documents, key-value pairs, wide-column families and graphs - and are designed to scale out across many commodity servers rather than up on one large machine. They trade some of SQL's strict schema and ACID guarantees for horizontal scalability, flexible schemas and high availability, as summarised by the CAP theorem and the BASE model. In Indian university syllabi NoSQL appears as the last unit of the DBMS course and as a full elective (Big Data / Advanced Databases); GATE CS asks little of it directly, but placement technical rounds at Flipkart, Swiggy, Zomato, Paytm, PhonePe, Jio, Razorpay, TCS, Infosys and Wipro ask it constantly. Typical questions are CAP, SQL-versus-NoSQL choice, MongoDB query syntax, sharding keys and when NOT to use NoSQL. Data engineers, backend developers and SRE roles use these systems daily behind UPI, e-commerce catalogues and ride-hailing apps.

Explained

Why NoSQL appeared, and what "NoSQL" means

• Relational databases assume a fixed schema, normalised tables and joins, and scale primarily by vertical scaling (a bigger server) - costly and bounded.
• Web-scale workloads (Google, Amazon, Facebook) needed horizontal scaling: hundreds of cheap nodes, data partitioned across them, tolerant of node failure. Joins across machines are expensive, so the data model changed instead.
• The founding papers are Google's Bigtable (2006) and Amazon's Dynamo (2007); open-source systems (HBase, Cassandra, MongoDB, Riak) followed.
• NoSQL = "not only SQL": non-relational, usually schema-less (schema-on-read), non-tabular, built for distribution and denormalised access. Many now add SQL-like query layers (Cassandra CQL, MongoDB aggregation, N1QL), so "no SQL syntax" is not the definition.
• Aggregate-oriented data model: document, key-value and column-family stores group all data an application reads together into one unit (the *aggregate*) identified by a key. One key lookup replaces a multi-table join. Graph databases are the exception - they are relationship-oriented.
• Schema-less does not mean no schema. The schema moves into application code; if the code assumes a field exists and old documents lack it, the code breaks. This is an implicit schema.

The four families (classification - asked almost every exam)

1. Key-value stores - a hash map: put(key, value), get(key), delete(key). Value is opaque (blob/string). Fastest and simplest; no querying inside the value.
   • Examples: Redis (in-memory, data structures: strings, lists, sets, sorted sets, hashes, streams), Amazon DynamoDB, Riak, Memcached (cache only, no persistence).
   • Uses: session store, cache, leaderboards, rate limiting, OTP store, shopping cart.
2. Document stores - value is a self-describing document (JSON / BSON / XML) that the database can index and query by inner fields; documents may nest arrays and sub-documents.
   • Examples: MongoDB, CouchDB, Couchbase, Elasticsearch (document + inverted-index search), Firestore.
   • Uses: product catalogues, user profiles, CMS, event logs, order documents.
3. Column-family / wide-column stores - rows keyed by a row key; columns grouped into column families; each row can have different columns; data stored column-family-wise on disk, so scans read only needed columns.
   • Examples: Apache Cassandra, HBase (on HDFS), Google Bigtable, ScyllaDB.
   • Uses: time-series and IoT telemetry, messaging history, write-heavy event logging, recommendation features.
4. Graph databases - nodes, edges (directed, typed, with properties) and properties; traversal is a pointer hop, not a join.
   • Examples: Neo4j (Cypher query language), JanusGraph, Amazon Neptune, ArangoDB (multi-model).
   • Uses: social graphs, fraud rings, recommendation ("friends of friends"), network/telecom topology, knowledge graphs.
5. Often listed as a fifth family: time-series databases (InfluxDB, TimescaleDB, Prometheus) and search engines; both are specialisations of the above.

CAP theorem, PACELC and BASE

• CAP theorem (Brewer, 2000; proved by Gilbert and Lynch, 2002): a distributed data store cannot simultaneously guarantee all three of
  • C - Consistency: every read sees the most recent committed write (this is *linearizability*, not the C of ACID).
  • A - Availability: every non-failing node returns a non-error response.
  • P - Partition tolerance: the system keeps working despite messages being lost between nodes.
• Because network partitions will happen, P is not optional in a distributed system: the real choice during a partition is C or A.
  • CP systems: HBase, MongoDB (default, single primary), Zookeeper, Google Spanner - refuse or block writes rather than diverge.
  • AP systems: Cassandra, DynamoDB, Riak, CouchDB - accept writes on both sides and reconcile later.
  • CA exists only in a single-node or non-partitioned system - not a real distributed option.
• PACELC (Abadi, 2012) extends CAP: if Partition, choose A or C; else (normal running) choose Latency or Consistency. Cassandra and Dynamo are PA/EL; Spanner and HBase are PC/EC. This explains why an AP store is also usually faster when nothing is broken.
• ACID vs BASE:
  • ACID - Atomicity, Consistency, Isolation, Durability (the relational contract).
  • BASE - Basically Available, Soft state (state may change without input, as replicas sync), Eventual consistency (if no new writes arrive, all replicas eventually converge).
• Consistency models (strongest to weakest): strict/linearizable → sequential → causal → read-your-writes / monotonic reads (session guarantees) → eventual.
• Modern NoSQL is not ACID-free: MongoDB 4.0+ supports multi-document ACID transactions, and single-document updates were always atomic; Cassandra offers lightweight transactions (Paxos-based compare-and-set). Treat "NoSQL has no transactions" as outdated.

Distribution: sharding, replication and quorum

• Sharding (horizontal partitioning): split rows/documents across nodes by a shard key.
  • Range sharding - contiguous key ranges; supports range scans but risks hotspots on sequential keys (timestamps, auto-increment IDs).
  • Hash sharding - hash the key; even distribution, no efficient range scan.
  • Consistent hashing - keys and nodes placed on a ring (0 to 2^m−1); a key belongs to the next node clockwise; adding or removing a node moves only ~K/N keys instead of rehashing everything. Virtual nodes (vnodes) give each physical node many ring positions to even out load. Used by Dynamo, Cassandra, Riak.
• Replication: every partition is stored on RF (replication factor) nodes, typically 3, placed in different racks or availability zones.
  • Leader-follower (master-slave): one primary takes writes, replicas serve reads - MongoDB replica sets, HBase region servers. Failover by election (MongoDB uses a Raft-like protocol).
  • Leaderless / multi-master: any replica accepts a write - Cassandra, Dynamo. Needs conflict resolution.
• Quorum / tunable consistency: with replication factor N, write to W replicas and read from R.
  • If R + W > N, read and write sets overlap, so a read sees the latest write - strong consistency.
  • If R + W ≤ N, reads may be stale - eventual consistency, but lower latency.
  • Cassandra levels: ONE, TWO, QUORUM (⌊N/2⌋+1), LOCAL_QUORUM (quorum in the local data centre), ALL, ANY.
• Conflict resolution: last-write-wins by timestamp (simple, can lose writes); vector clocks / version vectors to detect concurrent versions (siblings) and push the choice to the application; CRDTs (counters, sets) that merge deterministically.
• Repair mechanisms: hinted handoff (a live node holds a write for a down node), read repair (fix stale replicas during a read), anti-entropy with Merkle trees (compare hash trees to find diverging ranges).
• Storage engine: most NoSQL write paths use an LSM tree - append to a commit log, buffer in an in-memory memtable, flush to immutable SSTables, merge by compaction; Bloom filters avoid reading SSTables that cannot contain the key. This makes writes sequential and very fast, at the cost of read amplification. Relational engines mostly use B+ trees (good for reads and range scans, random writes).

Data modelling for NoSQL (query-first design)

• Relational modelling is data-first (normalise, then query). NoSQL is query-first: list the application's access patterns, then design one aggregate (or one Cassandra table) per query.
• Denormalisation and duplication are deliberate. Storage is cheap; a join across shards is not. The same value may be copied into several documents or tables - the application keeps them in step.
• Embedding vs referencing (document stores):
  • Embed when data is read together, is owned by the parent, and is bounded in size (order → order lines; post → last 20 comments).
  • Reference when the child is large, unbounded, shared, or updated independently (student → college). MongoDB's BSON document limit is 16 MB, so unbounded arrays must be referenced or bucketed.
• Cassandra primary key = partition key + clustering columns. The partition key decides the node; clustering columns decide the sort order within the partition. You can only query efficiently by the partition key (plus clustering prefix) - hence one table per query pattern.
• Anti-patterns: unbounded partitions (all rows in one partition key), hot partitions (date-only partition key for today's data), using a low-cardinality field (gender, status) as shard key, treating a document store as relational and doing application-side joins in a loop (N+1 queries).
• Indexes: secondary indexes exist (MongoDB B-tree indexes, including compound, multikey on arrays, text, geospatial, TTL; Cassandra SAI / materialised views) but in Cassandra a secondary index on a high-cardinality column scatters queries across all nodes - prefer a second denormalised table.
• MapReduce / aggregation: aggregate with MongoDB's aggregation pipeline ($match, $group, $sort, $lookup, $unwind) or with Spark/Hive over HBase and Cassandra; analytics usually runs outside the operational store.

Advantages, limitations and the NewSQL middle ground

• Advantages: linear horizontal scaling on commodity hardware; high write throughput; flexible/evolving schema (good for agile and semi-structured data); high availability with replication and no single point of failure (leaderless designs); natural fit for JSON over REST APIs; geo-distribution with local-quorum reads.
• Limitations: weaker consistency to reason about; limited or no cross-aggregate joins and ad-hoc queries; no universal standard query language (each system differs - skills are less portable); referential integrity and constraints enforced in application code, not the engine; multi-aggregate transactions limited or costly; data duplication means multi-place updates; operational complexity (tuning compaction, repairs, GC); reporting/BI tools expect SQL.
• When NOT to use NoSQL: strongly relational data with many-to-many joins, strict multi-row transactions (banking ledgers, double-entry accounting), complex ad-hoc reporting, and small datasets that fit comfortably on one server - PostgreSQL with a JSONB column often answers "we need flexible schema" without giving up ACID.
• NewSQL / distributed SQL: Google Spanner, CockroachDB, TiDB, YugabyteDB, Amazon Aurora, Vitess (MySQL sharding) - horizontal scale with SQL and ACID, using consensus (Paxos/Raft) and, in Spanner, TrueTime clocks. Also multi-model engines (ArangoDB, Cosmos DB, OrientDB) and polyglot persistence - using several stores in one product (Postgres for orders, Redis for sessions, Elasticsearch for search, Cassandra for events).
• Indian regulatory context: the Digital Personal Data Protection Act, 2023 governs personal data in any store (its Data Fiduciary duties and the erasure right apply from 13 May 2027 under the DPDP Rules 2025) - schema flexibility does not excuse missing purpose limitation, consent records or erasure ability; "right to erasure" is harder when data is duplicated across denormalised aggregates, so plan deletion paths. RBI's storage directive (2018) requires payment system data to be stored only in India, which drives data-centre/region choice for sharded clusters (operators should check the current circulars, as these rules are periodically updated). CERT-In directions require time-synchronised logs (commonly kept in time-series/log stores).

Key points

• NoSQL - non-relational, non-tabular, distribution-first stores; "not only SQL", not "never SQL".
• Four families - key-value, document, column-family (wide-column), graph; the first three are aggregate-oriented.
• Aggregate - the unit of data read, written and replicated atomically, identified by a key; replaces the join.
• Schema-on-read - the schema is enforced by the application, not the engine; an implicit schema still exists.
• CAP theorem - of consistency, availability and partition tolerance, only two are attainable; since P is unavoidable, the real choice under partition is C or A.
• CP vs AP - HBase/MongoDB/Spanner choose consistency; Cassandra/DynamoDB/Riak choose availability.
• PACELC - under Partition choose A or C, Else choose Latency or Consistency.
• BASE - Basically Available, Soft state, Eventual consistency; the counterpart of ACID.
• Eventual consistency - with no further writes, all replicas converge; stale reads are possible meanwhile.
• Quorum rule - R + W > N gives strong consistency; R + W ≤ N gives eventual consistency.
• Consistent hashing - ring placement so adding a node moves only about K/N keys; vnodes balance load.
• Sharding - range (scannable, hotspot-prone) vs hash (even, no range scans); the shard key decides scalability.
• Replication factor - copies per partition (usually 3), across racks/zones; leader-follower or leaderless.
• Conflict resolution - last-write-wins, vector clocks (siblings), or CRDTs.
• LSM tree + Bloom filter - write-optimised storage (commit log, memtable, SSTables, compaction) vs the relational B+ tree.
• Query-first modelling - design one aggregate or table per access pattern; denormalise on purpose.
• Embed vs reference - embed bounded, co-read data; reference unbounded or shared data (MongoDB doc limit 16 MB).
• NewSQL - Spanner, CockroachDB, TiDB, YugabyteDB: scale-out SQL with ACID via Raft/Paxos consensus.

Formulas and examples

Example 1 - Tunable consistency in Cassandra. A cluster has RF N = 3.

• QUORUM = ⌊3/2⌋ + 1 = 2.
• Write at QUORUM (W = 2), read at QUORUM (R = 2): R + W = 4 > 3 → strongly consistent; survives max(2,2) = 2 → 3 − 2 = 1 node failure.
• Write at ONE (W = 1), read at ONE (R = 1): R + W = 2 ≤ 3 → eventually consistent, lowest latency, tolerates 2 failures.
• Write at ALL (W = 3), read at ONE (R = 1): 3 + 1 = 4 > 3 → strongly consistent reads, but no write availability if any replica is down.
• With N = 5, QUORUM = 3; W = 3, R = 3 → 6 > 5, strong, tolerating 2 failures.

Example 2 - Capacity and node count for an IoT meter workload. A Bengaluru utility ingests readings from 2,00,000 smart meters, one reading every 15 minutes, each row 200 bytes, RF = 3, retention 2 years.

• Readings per meter per day = 24 × 60/15 = 96 → total/day = 2,00,000 × 96 = 1,92,00,000 rows/day.
• Raw bytes/day = 1.92 × 10⁷ × 200 = 3.84 × 10⁹ B = 3.84 GB/day; with RF 3 → 11.52 GB/day.
• Two years (730 days) = 11.52 × 730 ≈ 8,410 GB ≈ 8.41 TB replicated.
• Writes/second = 1.92 × 10⁷ / 86,400 ≈ 222 writes/s (×3 replicas ≈ 666 replica writes/s) - easily handled by a small cluster, so the sizing driver is disk, not throughput.
• At 2 TB usable per node and 60% target utilisation (headroom for compaction), usable/node = 1.2 TB → nodes = 8.41/1.2 ≈ 7.01 → round up to 8 nodes (7 would be just short, and the eighth leaves room for future growth).
• Partition key choice: (meter_id, month) with clustering column reading_time DESC - keeps a partition to 96 × 30 ≈ 2,880 rows instead of an unbounded 70,080 rows for 2 years.

Example 3 - Consistent hashing vs modulo hashing. K = 10,00,000 keys on N = 10 nodes; one node is added.

• Modulo hashing (hash(key) % N): N changes 10 → 11, so almost every key's target changes - roughly 10,00,000 × (1 − 1/11) ≈ 9,09,091 keys move (~91%).
• Consistent hashing: the new node takes only its arc of the ring ≈ K/(N+1) = 10,00,000/11 ≈ 90,909 keys move (~9%) - a 10× reduction in rebalancing traffic.

Example 4 - Document model for an Indian e-commerce order (MongoDB).

// orders collection - one document per order (embed line items: bounded, always read together)
{ _id: ObjectId("..."),
  orderNo: "ABL-2026-004821",
  userId: ObjectId("..."),            // reference: user profile updated independently
  placedAt: ISODate("2026-10-29T10:12:00Z"),
  status: "PACKED",
  payment: { mode: "UPI", vpa: "9xxxxxxxxx@ybl", amount: 2499, currency: "INR" },
  shipTo:  { line1: "Flat 4B, Rose Apt", city: "Kochi", state: "Kerala",
             pin: "682024", geo: { type: "Point", coordinates: [76.30, 9.96] } },
  items: [ { sku: "TSH-RED-M", name: "Cotton T-shirt", qty: 2, price: 699, gstPct: 5 },
           { sku: "SHO-BLK-9", name: "Running shoes", qty: 1, price: 1101, gstPct: 5 } ]
}

db.orders.createIndex({ userId: 1, placedAt: -1 })      // "my orders, newest first"
db.orders.createIndex({ "items.sku": 1 })               // multikey index on an array
db.orders.find({ userId: uid, status: { $in: ["PACKED","SHIPPED"] } })
         .sort({ placedAt: -1 }).limit(20)

// revenue per city for October 2026 (aggregation pipeline)
db.orders.aggregate([
  { $match: { placedAt: { $gte: ISODate("2026-10-01"), $lt: ISODate("2026-11-01") } } },
  { $unwind: "$items" },
  { $group: { _id: "$shipTo.city",
              revenue: { $sum: { $multiply: ["$items.price", "$items.qty"] } } } },
  { $sort: { revenue: -1 } } ])

• Order total check: 2 × 699 + 1 × 1101 = 1398 + 1101 = ₹2,499, matching payment.amount.

Example 5 - Same requirement, three models. "Show a user's last 20 orders."

Comparison table - SQL vs NoSQL

Comparison table - the four families

Where it is used

• E-commerce and quick commerce (Flipkart, Myntra, Meesho, BigBasket, Zepto): product catalogues with varying attributes in document stores, Redis for carts and sessions, Elasticsearch for search and faceting.
• Fintech and UPI (Paytm, PhonePe, Razorpay, Jupiter, NPCI ecosystem): Redis for rate limiting, idempotency keys and OTP; Cassandra/HBase for transaction event logs and audit trails; the money ledger itself stays on ACID SQL or NewSQL - a standard interview point.
• Telecom, utilities and IoT (Jio, Airtel, smart-meter and AMI projects, DISCOM analytics): Cassandra/HBase and time-series stores for call detail records and meter telemetry at crores of rows per day.
• Ride-hailing, food delivery and logistics (Ola, Uber India, Swiggy, Zomato, Delhivery): geospatial document queries for nearby partners, Redis geo-sets for live driver locations, graph stores for route/relationship analysis.
• Fraud, credit and identity graphs (banks, NBFCs, credit bureaus, insurance): Neo4j-style traversals to find shared devices, addresses or phone numbers linking fraudulent applications.
• Job roles: backend developer (MongoDB/Redis), data engineer (Cassandra, HBase, Spark, Kafka), SRE/DBA (replica sets, compaction, repairs, capacity), analytics engineer (aggregation pipelines, polyglot persistence design).

Interview questions

• Q: What does NoSQL actually mean, and is it anti-SQL? A: It means "not only SQL" - non-relational, usually schema-less stores built to scale horizontally; many expose SQL-like languages (CQL, N1QL), so it is a different data model, not a rejection of query languages.
• Q: State the CAP theorem and explain the real-world choice. A: A distributed store cannot guarantee consistency, availability and partition tolerance together. Partitions are unavoidable, so the design choice is consistency (CP: HBase, MongoDB) or availability (AP: Cassandra, DynamoDB) when a partition occurs.
• Q: Is the C in CAP the same as the C in ACID? A: No. CAP's C is linearizability - every read sees the latest write; ACID's C is that a transaction leaves the database satisfying its integrity constraints.
• Q: What is eventual consistency, and how do you get strong consistency on an eventually consistent store? A: Replicas converge once writes stop, so reads may be stale; choosing read and write quorums such that R + W > N makes the sets overlap and gives strongly consistent reads.
• Q: With RF = 3, which Cassandra consistency levels give strong consistency? A: Any pair with R + W > 3 - for example QUORUM write (2) with QUORUM read (2), or ALL write with ONE read.
• Q: When would you choose MongoDB over Cassandra? A: When you need rich ad-hoc queries, secondary indexes and aggregations on flexible documents with a single primary; Cassandra wins for very high write throughput, multi-datacentre availability and known key-based access patterns.
• Q: How do you choose a shard or partition key? A: Pick a high-cardinality, evenly distributed field that also appears in the common query so reads hit one shard - avoid monotonically increasing keys (timestamps, auto IDs) that create hotspots, and avoid low-cardinality fields that create huge partitions.
• Q: Embed or reference in a document database? A: Embed when the child data is bounded, owned by the parent and read together; reference when it is unbounded, large or shared - also because a MongoDB document cannot exceed 16 MB.
• Q: Why are NoSQL writes often faster than relational writes? A: LSM-tree engines append to a commit log and an in-memory memtable and flush sequentially to immutable SSTables, avoiding random in-place B+ tree updates; the cost is compaction and read amplification.
• Q: Does NoSQL support transactions? A: Single-aggregate (document/row) updates are atomic everywhere; MongoDB 4.0+ supports multi-document ACID transactions, and Cassandra has Paxos-based lightweight transactions - but cross-shard transactions remain comparatively expensive.
• Q: Give a case where NoSQL is the wrong choice. A: A core banking ledger or accounting system needing multi-row ACID transactions, foreign keys and ad-hoc reporting - a relational or NewSQL (Spanner, CockroachDB) store is correct.
• Q: What is polyglot persistence? A: Using several databases in one application, each for the job it does best - Postgres for orders, Redis for sessions, Elasticsearch for search, Cassandra for event history.
• Q: What is consistent hashing and why does Cassandra use vnodes? A: Keys and nodes are mapped onto a hash ring so a membership change relocates only about K/N keys; vnodes give each node many small ring ranges, so load and streaming during repair or scale-out stay balanced.

Common mistakes

• Saying "NoSQL is faster than SQL" as a blanket claim - it is faster for key-based access at scale and usually slower or incapable for joins, ad-hoc queries and aggregates.
• Claiming CAP lets you "pick any two" - you cannot drop partition tolerance in a distributed system, so you are really choosing C or A under partition.
• Confusing CAP consistency with ACID consistency, or calling every NoSQL store "AP" (MongoDB and HBase are CP).
• Treating "schema-less" as "no design needed", then writing defensive code everywhere for missing fields; the schema just moved into the application.
• Modelling in third normal form on a document or Cassandra store and then doing joins in application loops (N+1 reads).
• Picking a shard key by convenience (timestamp, status, auto-increment id) and creating hotspots or unbounded partitions.
• Repeating that NoSQL has no transactions or no indexes - both are outdated for current MongoDB, Cassandra and DynamoDB.
• Forgetting the replication factor when sizing storage, or ignoring compaction headroom (plan for roughly 50-70% disk utilisation).
• Assuming flexible schemas exempt you from compliance - DPDP Act 2023 duties such as erasure and purpose limitation still apply to duplicated, denormalised copies.

Related topics

Transactions, ACID and indexing · ER model and normalisation · Query processing and optimisation · Database design case studies · Distributed systems and consensus (Paxos, Raft) · Big data processing with Hadoop and Spark