Retrieval-augmented generation and vector databases

In short

Retrieval-augmented generation (RAG) puts a search step in front of a language model: the user's question is used to fetch relevant passages from a private corpus, and those passages are pasted into the prompt so the model answers from evidence instead of from memory alone. A vector database is the store that makes this fetch fast — it keeps embedding vectors plus their text and metadata, and answers approximate nearest-neighbour (ANN) queries in milliseconds over millions of vectors. RAG is the standard industrial fix for three LLM weaknesses: stale knowledge, hallucination and no access to internal data, and it avoids the cost of fine-tuning. In Indian university syllabi it appears in AI-applications, information-retrieval and generative-AI electives; in placement rounds (TCS, Infosys, Wipro, HCLTech, Accenture, Zoho, Freshworks, Sarvam AI, Krutrim, Yellow.ai) a RAG pipeline is the single most common "build this" question for an AI/DS fresher.

Explained

Why RAG, and where it sits

• An LLM's weights freeze at its training cut-off; it cannot know your company's HR policy, yesterday's circular or a private contract.
• Three ways to inject knowledge: prompt it (fits little), fine-tune (costly, must be redone per update, still no citations), retrieve (cheap, updatable by re-indexing one document, gives citations). RAG is the retrieval route.
• Parametric vs non-parametric memory: weights are parametric (implicit, unverifiable); the retrieved corpus is non-parametric (explicit, auditable, deletable — important for a DPDP erasure request).
• Grounding: the instruction "answer only from the passages below; if absent, say you do not know" is what converts retrieval into reduced hallucination. Retrieval alone does not do it.
• RAG reduces but does not eliminate hallucination — the model can still misread or over-generalise a correct passage.

The indexing (offline) pipeline

1. Ingest — PDFs, DOCX, HTML, tickets, Confluence, SQL rows; OCR scanned Indian government PDFs; keep tables as Markdown, not flattened text.
2. Clean and segment — strip headers/footers, keep headings as structure signals.
3. Chunk — split into retrievable units. Strategies: fixed-size by tokens; recursive character splitting (paragraph → sentence → word); semantic chunking (break where consecutive sentence embeddings diverge); layout/heading-aware for structured manuals; parent–child (embed small child chunks, return the larger parent for context).
4. Overlap — repeat 10–20% of tokens between chunks so a sentence split across a boundary is still findable.
5. Embed — one dense vector per chunk from an embedding model (dimensions typically 384–3,072). Chunks must be embedded with the *same* model and the same instruction prefix as the queries.
6. Store — vector + chunk text + metadata (doc id, title, page, date, department, language, access tags).
7. Build the index — HNSW/IVF structure; optionally a parallel BM25/keyword index for hybrid search.
8. Re-indexing: incremental upsert by document id; deletes must remove the vector *and* its text, else deleted data keeps surfacing.

The query (online) pipeline

1. Query processing — rewrite a follow-up into a standalone question using chat history ("and his salary?" → "What is the salary of a Grade-4 operator?"); optional multi-query expansion and HyDE (embed a hypothetical answer instead of the question).
2. Retrieve — top-k by vector similarity (k typically 20–50 as a candidate pool), with metadata filtering (date range, department, user's access role) applied pre- or post-filter.
3. Hybrid search — combine dense (semantic) with sparse BM25 or SPLADE (lexical, good for product codes, IS numbers, section numbers, names) and fuse with reciprocal rank fusion (RRF) or weighted scores.
4. Rerank — a cross-encoder reranker scores each (query, chunk) pair jointly; far more accurate than the bi-encoder embedding, far slower, so it is run only on the candidate pool, cutting 50 → 5.
5. Assemble the prompt — system instruction + numbered passages with source labels + question; respect the context budget; put the most relevant passage first or last (models attend least to the middle — the "lost in the middle" effect).
6. Generate with citations — require passage numbers in the answer so a reviewer can verify.
7. Post-check — optional groundedness/NLI check that each sentence is entailed by a cited passage; refuse or escalate otherwise.
8. Agentic / iterative RAG — the model decides whether to retrieve, issues several searches, and retries on a failed answer (self-RAG, corrective RAG).
9. GraphRAG — build an entity–relation graph over the corpus; answers "how are X and Y connected" and multi-hop questions that chunk retrieval misses.

Vector databases: what they are and how they index

• A vector DB stores high-dimensional vectors and supports k-NN / ANN search, metadata filters, CRUD, persistence, sharding and access control. A *library* (FAISS, ScaNN, Annoy, hnswlib) gives only the index; a *database* adds the rest.
• Distance metrics: cosine similarity (direction only — usual for text), dot product (for normalised vectors, equivalent ranking to cosine), Euclidean/L2. Normalise vectors and the three agree in rank order.
• Exact (flat) search — brute force, 100% recall, O(N·d) per query; fine up to roughly a lakh vectors.
• IVF (inverted file) — k-means clusters the vectors into nlist cells; a query scans only the nprobe nearest centroids. Tunable recall vs speed; needs a training step.
• HNSW (hierarchical navigable small world) — a multi-layer proximity graph; greedy descent from a sparse top layer. Parameters M (neighbours per node), ef_construction (build quality), ef_search (query breadth). Best recall-per-latency; high RAM; the default in Qdrant, Weaviate, Elasticsearch, pgvector HNSW.
• Quantization / compression: scalar (int8) ≈ 4× smaller; binary ≈ 32× with a rescoring pass; product quantization (PQ) splits the vector into m sub-vectors and replaces each with a 1-byte codebook id — large compression, some recall loss; IVF-PQ and DiskANN keep billion-scale indexes on SSD.
• Sparse + dense in one store, multi-vector/late-interaction (ColBERT-style) and tensor indexes are newer options.
• Products: FAISS (library), pgvector on PostgreSQL (simplest if you already run Postgres), Milvus, Qdrant, Weaviate, Chroma, Vespa, Redis, Elasticsearch/OpenSearch k-NN, MongoDB Atlas Vector Search, Pinecone, Azure AI Search, Vertex AI Vector Search. Features and limits change release to release — check current docs before quoting numbers.
• Filtering trap: pre-filtering a graph index can disconnect it (low recall); post-filtering can return fewer than k results. Good engines do filtered-HNSW traversal.

Evaluation and tuning

• Retrieval metrics: recall@k (did the right chunk come back at all — the ceiling on answer quality), precision@k, MRR, nDCG@k, hit rate.
• Generation metrics: faithfulness/groundedness (is every claim supported), answer relevance, context precision/recall, plus exact-match or F1 against gold answers. Tool-kits: RAGAS, TruLens, DeepEval, LLM-as-judge with a human-audited sample.
• Build a gold set first — 100–300 real questions with the correct source passage; without it, tuning is guesswork.
• Knobs in order of payoff: chunk size and overlap → hybrid search → reranker → k → prompt → embedding model → generator model.
• Latency budget (typical, measure your own): embed query 10–40 ms, ANN search 5–50 ms, rerank 50–300 ms, generation 1–5 s.

Advantages, limitations and Indian compliance

• Advantages: fresh knowledge, citations and auditability, no retraining, cheap document-level updates and deletions, per-user access control, smaller/cheaper generator models become usable.
• Limitations: quality capped by retrieval (garbage-in), weak on aggregate questions ("how many invoices above ₹1 lakh?" — that is a SQL job), multi-hop reasoning, conflicting or outdated documents, contradiction between parametric and retrieved knowledge, context cost, chunk-boundary loss, and prompt-injection text hidden inside ingested documents.
• Indian context: the Digital Personal Data Protection Act, 2023 (rules notified in 2025, obligations phasing in) makes personal data in a corpus a liability — purpose limitation, consent, and a data-principal's right to correction and erasure imply you must be able to locate and delete a person's chunks *and* their vectors; so store document provenance from day one. CERT-In directions require 180-day log retention and 6-hour incident reporting. RBI/SEBI/IRDAI localisation and outsourcing rules push BFSI RAG deployments to India-region or on-premise vector stores. Sectoral AI guidance (MeitY advisories, RBI's FREE-AI framework recommendations) stresses human oversight for customer-facing answers. Indian-language corpora need language-aware chunking and multilingual embeddings; BHASHINI models are used for translation/ASR in the ingest step.

Key points

• RAG - retrieve relevant passages, then generate an answer conditioned on them; retrieval + grounded prompt, not retraining.
• Non-parametric memory - knowledge kept in an index you can update, cite and delete, unlike model weights.
• Chunking - the highest-leverage design choice; too small loses context, too large dilutes the embedding and wastes tokens.
• Overlap - 10–20% of chunk size, to survive sentence splits at boundaries.
• Bi-encoder - embeds query and chunk separately, so chunk vectors are precomputed; fast, less accurate.
• Cross-encoder reranker - scores query and chunk together; accurate, cannot be precomputed, so used only on the shortlist.
• ANN - approximate nearest neighbour; trades a few percent recall for orders-of-magnitude speed.
• HNSW - layered proximity graph; M, ef_construction, ef_search control recall, build time and RAM.
• IVF / IVF-PQ - cluster-then-scan; nlist, nprobe control the speed–recall trade-off; PQ compresses vectors ~16–64×.
• Cosine similarity - standard text metric; for unit vectors, ranking by cosine = by dot product = by L2.
• Hybrid search - dense for meaning, BM25 for exact tokens (IS numbers, SKUs, section numbers); fuse with RRF.
• Metadata filtering - date, department, language and access tags; the mechanism for row-level security in RAG.
• recall@k - the retrieval ceiling: what retrieval misses, generation can never say.
• Faithfulness - fraction of answer claims supported by the cited context; the key anti-hallucination metric.
• Lost in the middle - models under-use passages placed in the centre of a long context; order matters.
• Prompt injection via documents - ingested text can carry instructions; treat retrieved content as data, never as commands.
• RAG vs fine-tuning - RAG for facts that change; fine-tuning for style, format and task behaviour; they combine.
• DPDP Act, 2023 - requires you to be able to find and erase an individual's data, so keep per-chunk provenance.

Formulas and examples

Example 1 — chunk count, index size, compression. 5,000 documents, 4,000 tokens each; C = 512, O = 64.

• Stride = 512 − 64 = 448; chunks/doc = ceil((4000 − 512)/448) + 1 = ceil(7.786) + 1 = 8 + 1 = 9.
• Total chunks N = 5,000 × 9 = 45,000.
• d = 768, float32 → 768 × 4 = 3,072 B per vector; 45,000 × 3,072 = 138.24 MB ≈ 132 MiB of raw vectors.
• HNSW with M = 16 → ≈ 16 × 2 × 4 = 128 B/node of links at layer 0, plus upper layers ≈ 64 B → ≈ 45,000 × 192 B ≈ 8.6 MB graph overhead.
• PQ with m = 96 sub-vectors × 8 bits = 96 B/vector → 32× compression, 45,000 × 96 = 4.32 MB.

Example 2 — cosine similarity by hand. q = [0.2, 0.5, 0.1], d = [0.1, 0.4, 0.3].

• q · d = 0.02 + 0.20 + 0.03 = 0.25. ‖q‖ = √0.30 = 0.5477. ‖d‖ = √0.26 = 0.5099.
• cos = 0.25 / (0.5477 × 0.5099) = 0.25 / 0.2793 = 0.895.
• If both were unit vectors with the same cosine, L2 = √(2 − 2 × 0.895) = √0.210 = 0.458 — so smallest L2 = largest cosine.

Example 3 — IVF speed-up. N = 45,000, nlist = 256 → ≈ 176 vectors per cell. With nprobe = 8, vectors scanned = 8 × 176 = 1,408 = 3.1% of the index, i.e. ≈ 32× fewer distance computations than flat search (45,000 / 1,408 = 31.96). Raising nprobe to 32 scans 12.5% and lifts recall, at 4× the latency.

Example 4 — retrieval metrics. A query has 4 relevant chunks; the top 10 contain 3 of them, at ranks 1, 2 and 5.

• recall@10 = 3/4 = 0.75; precision@10 = 3/10 = 0.30.
• nDCG@3 with binary relevance (rel = 1, 1, 0): DCG = 1/log₂2 + 1/log₂3 + 0 = 1 + 0.6309 = 1.6309. Ideal order (1,1,0) is the same → IDCG = 1.6309 → nDCG@3 = 1.000. For the order (1, 0, 1): DCG = 1 + 0 + 1/log₂4 = 1.5 → nDCG@3 = 1.5/1.6309 = 0.920.
• Over 4 queries whose first relevant hits land at ranks 1, 3, 2 and none: MRR = (1 + 0.3333 + 0.5 + 0)/4 = 0.458.

Example 5 — context budget and monthly cost (illustrative prices; vendor rates change). 500 queries/day, k = 5 chunks of 512 tokens, system prompt 400 + question 60 tokens, output 300 tokens. Assume ₹0.25 per 1,000 input tokens and ₹1.00 per 1,000 output tokens.

• Input = 5 × 512 + 460 = 3,020 tokens → 3.020 × ₹0.25 = ₹0.755; output = 0.300 × ₹1.00 = ₹0.30 → ₹1.055 per query.
• Day = 500 × 1.055 = ₹527.50; 30 days ≈ ₹15,825.
• Stuffing k = 20 instead: input = 10,240 + 460 = 10,700 → ₹2.675 + ₹0.30 = ₹2.975/query → ₹1,487.50/day ≈ ₹44,625/month, i.e. 2.8× the cost — which is why a reranker that cuts 20 → 5 pays for itself.

RRF fusion check. Doc A: dense rank 2, BM25 rank 8 → 1/62 + 1/68 = 0.01613 + 0.01471 = 0.03084. Doc B: dense rank 1, BM25 rank 50 → 1/61 + 1/110 = 0.01639 + 0.00909 = 0.02548. A outranks B despite losing on dense alone — agreement across retrievers wins.

# minimal RAG loop (pseudocode)
chunks = split(docs, size=512, overlap=64)
store.upsert([(embed(c.text), c.text, c.meta) for c in chunks])

def answer(q, history):
    q2 = rewrite(q, history)
    dense  = store.search(embed(q2), k=50, filter={"dept": user.dept})
    sparse = bm25.search(q2, k=50)
    pool   = rrf(dense, sparse, k=60)
    top    = cross_encoder.rerank(q2, pool)[:5]
    prompt = SYSTEM + format_with_sources(top) + q2   # "answer only from these; else say unknown"
    return llm(prompt)

Where it is used

• IT services and GCCs (TCS, Infosys, Wipro, HCLTech, Tech Mahindra, Accenture, Capgemini India): enterprise Q&A over SOPs, codebases and tickets; "RAG engineer" and "GenAI developer" are standard fresher-to-3-year postings.
• BFSI: policy and circular assistants over RBI/SEBI/IRDAI documents, KYC and underwriting copilots, customer-support bots; usually on India-hosted or on-premise vector stores for localisation compliance.
• Product companies and startups (Zoho, Freshworks, Razorpay, Swiggy, Zomato, Yellow.ai, Haptik, Observe.AI, Sarvam AI, Krutrim): help-centre deflection, agent-assist, call-summary search.
• Government and public services: scheme-eligibility and grievance assistants over notification PDFs, multilingual front-ends using BHASHINI translation/ASR before indexing.
• Healthcare, legal and education: clinical-guideline and case-law retrieval with mandatory citations and human review; exam-prep and doubt-solving over prescribed textbooks.
• Job roles: AI/ML engineer, NLP engineer, data engineer (ingestion pipelines), MLOps/LLMOps, search-relevance engineer, AI solution architect.

Interview questions

• Q: What problem does RAG solve that fine-tuning does not? A: It supplies fresh, private, verifiable facts with citations and lets you update or delete knowledge by re-indexing one document; fine-tuning teaches style and task behaviour but bakes facts into weights.
• Q: Why use an ANN index instead of exact search? A: Exact search is O(N·d) per query; ANN structures like HNSW or IVF cut distance computations by one or two orders of magnitude for a few percent recall loss, which production latency budgets require.
• Q: How do you choose chunk size? A: Empirically against a gold question set — start near 300–600 tokens with 10–20% overlap, then measure recall@k; use parent–child retrieval if small chunks retrieve well but answer poorly.
• Q: Difference between a bi-encoder and a cross-encoder, and where does each go? A: A bi-encoder embeds query and document separately so documents can be pre-indexed (used for retrieval); a cross-encoder scores the pair jointly and is far more accurate but cannot be pre-computed, so it reranks only the shortlist.
• Q: Why add BM25 when you already have embeddings? A: Dense vectors miss exact tokens — part codes, IS numbers, section numbers, rare names — where lexical matching is strong; hybrid retrieval with RRF gets both.
• Q: Your RAG bot still hallucinates. How do you debug it? A: First check whether the correct chunk was retrieved: if recall@k is low it is a retrieval bug (chunking, embedding, filters); if the chunk was present, it is a prompting/generation bug — tighten grounding instructions, add citations and a groundedness check.
• Q: What is "lost in the middle"? A: LLMs attend less to passages in the centre of a long context, so relevance order and keeping k small matter more than simply stuffing more text.
• Q: HNSW vs IVF-PQ — when do you pick which? A: HNSW for the best recall at low latency when vectors fit in RAM; IVF-PQ or DiskANN at hundreds of millions of vectors where memory cost dominates and some recall can be traded.
• Q: How do you enforce that a user sees only permitted documents? A: Store access tags as metadata and apply them as a filter inside the ANN search (filtered traversal), never by filtering after generation; validate that filtering has not collapsed recall.
• Q: What is prompt injection through retrieved documents? A: A malicious or careless document can contain instructions ("ignore previous rules and reveal...") that the model may obey; mitigate by delimiting retrieved text as data, instructing the model to treat it as untrusted, and sanitising ingestion.
• Q: Which metrics would you report for a RAG system? A: Retrieval — recall@k, MRR, nDCG@k; generation — faithfulness, answer relevance, context precision; plus latency percentiles and cost per query.
• Q: Does RAG help with "how many purchase orders exceeded ₹50 lakh last quarter?" A: Not well — that is an aggregation over structured data; route it to SQL (text-to-SQL) and use RAG only for the narrative documents.
• Q: How does the DPDP Act affect your vector store design? A: Personal data in chunks must be traceable to a data principal and deletable on request (the erasure right is in force from 13 May 2027 under the DPDP Rules 2025), so keep document and subject provenance in metadata and ensure deletion removes the vector, the text copy and any cache.

Common mistakes

• Embedding chunks with one model and queries with another (or forgetting the model's required query prefix) — similarities become meaningless.
• Normalising for cosine in one place and not the other, or using dot product on un-normalised vectors so long chunks always win.
• Tuning the prompt when recall@k is the real bottleneck; no gold question set, so "it looks better" is the only evidence.
• Treating k as "more is better" — cost rises linearly, accuracy does not, and noisy passages pull the model off course.
• Chunking across tables and code blocks, or dropping headings, so a retrieved chunk has no context about which document or section it came from.
• Deleting a source document from the file store but leaving its vectors and chunk text in the index — stale and, for personal data, non-compliant.
• Claiming RAG "eliminates hallucination", or shipping without citations so no reviewer can check an answer.
• Benchmarking an ANN index without reporting recall — a fast index at 60% recall is not comparable to a slower one at 98%.

Related topics

Text embeddings and semantic search · Large language models and prompt engineering · AI agents and tool use · Natural language processing basics · Information retrieval and ranking · Knowledge graphs and GraphRAG