Text embeddings and semantic search

In short

A text embedding is a fixed-length vector of real numbers that represents the meaning of a word, sentence or document, learned so that texts with similar meaning land close together in the vector space. Semantic search uses these vectors: the query is embedded, and documents are ranked by vector similarity instead of by exact keyword match, so "two-wheeler loan EMI" can retrieve "bike finance instalment". Embeddings are the measurement layer under search, recommendation, deduplication, classification and clustering, and they are asked in AI/DS university papers, GATE DA (vector representations, similarity, evaluation metrics) and in almost every applied-AI interview at Indian employers. Typical production work is choosing an embedding model, chunking text, computing cosine similarity, and tuning recall against latency and cost.

Explained

What an embedding is

• Definition: a mapping E: text → R^d that turns a token, sentence or passage into a dense vector of d real numbers (commonly d = 128, 384, 768, 1024, 1536).
• Dense vs sparse: one-hot and bag-of-words/TF-IDF vectors are sparse (length = vocabulary, mostly zeros, no notion of similarity between distinct words); embeddings are dense, low-dimensional and carry graded similarity.
• Distributional hypothesis (Firth): words occurring in similar contexts have similar meanings — the assumption every embedding method encodes.
• Semantic geometry: direction carries meaning, so vector arithmetic works approximately: king − man + woman ≈ queen; Delhi − India + Japan ≈ Tokyo.
• Embedding matrix: for a vocabulary V, a V × d matrix whose rows are the learned word vectors; look-up is a row index, which is why it is cheap.
• Properties wanted: isotropy (vectors spread over the space), alignment (paraphrases close), uniformity (unrelated texts far), and a bounded similarity range after L2 normalisation.

How embeddings are learned

• Count-based: build a word-context co-occurrence matrix, weight by PPMI (positive pointwise mutual information), and reduce with SVD/LSA → latent semantic vectors.
• word2vec (Mikolov, 2013): a shallow 2-layer network. CBOW predicts the centre word from the average of context words (fast, good for frequent words); skip-gram predicts each context word from the centre word (better for rare words and small corpora).
• Negative sampling: instead of a softmax over the whole vocabulary, train a logistic classifier to separate one true (word, context) pair from k random negatives (k ≈ 5–20) — turns an O(V) cost into O(k).
• Subsampling and window: very frequent words (the, is) are dropped with probability based on frequency; window size 2–10 — small windows capture syntax, large windows capture topic.
• GloVe (2014): factorises the log co-occurrence matrix with a weighted least-squares objective — global statistics rather than local windows.
• fastText: a word vector is the sum of character n-gram vectors, so out-of-vocabulary and morphologically rich words (Hindi, Tamil, Malayalam inflections, misspellings in Indian user queries) still get sensible vectors.
• Static vs contextual: word2vec/GloVe/fastText give one vector per word type — "bank" has a single vector. Transformer encoders (BERT and successors) give a different vector per occurrence, resolving polysemy from context.
• Sentence embeddings: a transformer's token vectors must be pooled — mean pooling (usual best), CLS-token pooling, or max pooling. Raw BERT CLS vectors are poor for similarity; Sentence-BERT (SBERT) fine-tunes with a siamese objective so cosine similarity becomes meaningful.
• Contrastive training: given (query, positive passage) pairs and in-batch negatives, minimise InfoNCE loss so positives score higher than negatives; hard negatives (plausible but wrong passages, mined with BM25) matter more than more data.
• Matryoshka / truncatable embeddings: trained so the first 64/128/256 dimensions are usable alone, letting you trade accuracy for storage.

Similarity, distance and the geometry

• Cosine similarity measures angle only, ignoring length — the default for text, because document length should not change topical similarity. Range −1 to +1 (0 to 1 in practice for most modern encoders).
• Dot product (inner product) rewards both angle and magnitude; some models (trained for MIPS, maximum inner product search) expect dot product — using cosine on them loses calibrated magnitude.
• Euclidean (L2) distance: after L2 normalisation, ranking by L2 distance and by cosine is identical, since ‖a − b‖² = 2 − 2cos θ. This is why most indexes normalise first.
• Curse of dimensionality: in high d, distances concentrate; nearest-neighbour contrast weakens, so metrics must match the training objective, and raw d larger than needed wastes memory without gain.
• Dimensionality reduction for inspection: PCA (linear, preserves variance), t-SNE and UMAP (non-linear, for 2-D plots only — never for retrieval, as global distances are distorted).

The semantic search pipeline

1. Chunk: split documents into retrievable units (typically 200–500 tokens with 10–20% overlap); respect headings so a chunk is self-contained. Over-long chunks dilute meaning; over-short chunks lose context.
2. Embed (offline): run every chunk through the encoder, L2-normalise, store vector + text + metadata (source, date, language, access tag).
3. Index: exact flat index (brute force, 100% recall) for small corpora; approximate nearest neighbour for large ones — IVF (cluster, probe nprobe lists), HNSW (navigable small-world graph; parameters M, efConstruction, efSearch), PQ/IVF-PQ (product quantisation, heavy compression), ScaNN.
4. Query: embed the query with the same model and same normalisation, retrieve top-k (k = 50–200 for reranking), optionally filter by metadata.
5. Rerank: a cross-encoder scores (query, passage) jointly and is far more accurate but O(k) forward passes — so it is applied only to the shortlist. The retriever is a bi-encoder (independent, precomputable, fast).
6. Hybrid: combine dense scores with lexical BM25 (good at rare words, codes, product numbers like "HSN 8471") using weighted sum or Reciprocal Rank Fusion (RRF): score = Σ 1/(60 + rank).
7. Asymmetric search: short query vs long passage — use a model trained for it, often with instruction prefixes ("query: ", "passage: "). Symmetric search (duplicate questions) uses the same prefix for both.

Evaluation and tuning

• Offline metrics: Recall@k and Hit Rate@k (is a relevant item retrieved at all), Precision@k, MRR (reciprocal rank of the first relevant hit), MAP, nDCG@k (graded relevance with position discount), and latency (p50/p95) and cost per 1,000 queries.
• Benchmarks: MTEB (Massive Text Embedding Benchmark) ranks models on retrieval, clustering, classification and STS; BEIR for zero-shot retrieval. Use them to shortlist, then test on your own labelled queries — domain shift is large.
• Build a gold set: 100–300 real queries with judged relevant documents; measure before and after every change (model, chunk size, hybrid weight, reranker).
• Semantic textual similarity (STS): correlation (Spearman) between predicted cosine and human similarity scores — the classic sentence-embedding evaluation.
• Thresholds are model-specific: a cosine of 0.75 may mean "near duplicate" for one model and "vaguely related" for another; calibrate on your own data instead of copying a number.
• Cost drivers: one-time embedding of the corpus, re-embedding on every model change (the big hidden cost), vector storage (d × 4 bytes per vector at float32), and query-time embedding.

Limits, Indian-language practice and compliance

• No reasoning: an embedding measures similarity, not truth or logic; negation ("loan approved" vs "loan not approved") and numbers/dates are poorly separated by most encoders.
• Token and domain limits: every encoder has a max sequence length (commonly 256–512 tokens; some 8k) — text beyond it is silently truncated. Jargon, Indian abbreviations and code-mixed text need domain fine-tuning.
• Bias: embeddings inherit caste, gender, religion and region stereotypes from the training corpus; measurable with association tests (WEAT) and reducible but not eliminable.
• Indic/multilingual options: multilingual encoders (LaBSE, multilingual-E5, mUSE) place Hindi and English paraphrases close; MuRIL (Google) and IndicBERT/IndicNLPSuite (AI4Bharat, IIT Madras) are trained on Indian languages with transliterated text, which matters for Hinglish queries. Transliteration ("ghar ka loan") is a genuine production problem.
• Data protection: embeddings of personal data are still personal data — the Digital Personal Data Protection Act, 2023 (with Rules notified in November 2025; these duties apply from 13 May 2027) governs purpose limitation, consent, retention and breach reporting; sectoral rules add to it (RBI's directions on storing payment and customer data in India, SEBI/IRDAI circulars). Embedding inversion attacks can partially reconstruct source text, so treat vector stores as sensitive, access-controlled stores — not anonymised data.
• Also relevant: IT Act, 2000 with the SPDI Rules, 2011 for reasonable security practices until fully displaced by DPDP implementation.

Key points

• Embedding - a dense fixed-length vector representation of text where distance encodes semantic similarity.
• Distributional hypothesis - meaning is learned from the company a word keeps.
• Sparse vs dense - TF-IDF/one-hot are vocabulary-sized and have no similarity between distinct terms; embeddings are low-dimensional and graded.
• CBOW vs skip-gram - predict centre from context (faster) vs context from centre (better for rare words).
• Negative sampling - replaces a V-way softmax with k+1 binary decisions, making word2vec trainable at scale.
• fastText - subword n-grams give vectors for unseen and inflected words.
• Static vs contextual - one vector per word type vs one vector per occurrence; only contextual models disambiguate polysemy.
• Pooling - mean pooling of token vectors is the standard way to get a sentence vector; raw CLS is weak without fine-tuning.
• SBERT / contrastive training - siamese fine-tuning with hard negatives is what makes cosine similarity usable.
• Cosine vs dot product - cosine ignores magnitude; use whichever the model was trained with.
• L2-normalised equivalence - cosine ranking equals Euclidean ranking once vectors are unit length.
• Bi-encoder vs cross-encoder - precomputable and fast vs jointly-scored and accurate; retrieve with the first, rerank with the second.
• ANN index - HNSW/IVF/PQ trade a little recall for orders-of-magnitude speed; flat index gives exact results.
• Hybrid search - dense + BM25 (often fused by RRF) beats either alone, especially for codes, names and rare terms.
• Metrics - Recall@k, MRR, nDCG@k, plus p95 latency and cost.
• Same model both sides - query and corpus must use one encoder, one normalisation, one prefix convention.
• Storage - float32 costs 4d bytes per vector; int8 cuts it 4× and binary 32× with modest recall loss.
• Embeddings are not anonymous - they are personal data under the DPDP Act, 2023 and are partially invertible.

Formulas and examples

Example 1 — cosine, distance and their relation. a = [2, 1, 0, 3], b = [1, 0, 1, 2].

• a · b = 2(1) + 1(0) + 0(1) + 3(2) = 2 + 0 + 0 + 6 = 8
• ‖a‖ = √(4 + 1 + 0 + 9) = √14 = 3.7417; ‖b‖ = √(1 + 0 + 1 + 4) = √6 = 2.4495
• cos θ = 8 / (3.7417 × 2.4495) = 8 / 9.1652 = 0.873 → θ ≈ 29.2°
• ‖a − b‖² = ‖a‖² + ‖b‖² − 2(a·b) = 14 + 6 − 16 = 4 → distance = 2.0 (check: a − b = [1, 1, −1, 1], √4 = 2 ✓)
• Note the dot product (8) alone would rank a long document above a short one of equal topical fit — hence cosine.

Example 2 — normalisation makes metrics interchangeable. Two unit vectors have cos θ = 0.90.

• ‖a − b‖² = 2 − 2(0.90) = 0.20 → ‖a − b‖ = 0.447
• A second pair with cos θ = 0.60 gives ‖a − b‖ = √(2 − 1.2) = √0.8 = 0.894. Higher cosine ⇒ smaller distance, so the ranking order is the same either way.

Example 3 — TF-IDF vs embedding behaviour. Corpus N = 10,000 documents; the term "subvention" appears in df = 50 documents; a document of 100 words uses it 3 times.

• tf = 3/100 = 0.03; idf = log₁₀(10,000/50) = log₁₀ 200 = 2.301
• w = 0.03 × 2.301 = 0.069 — a high weight, so BM25/TF-IDF finds this rare term well.
• But a query "interest subsidy scheme" shares no term with it, so lexical score = 0 while a dense encoder still scores it high. This is exactly the case hybrid search is built for.

Example 4 — retrieval evaluation on 3 queries. Rank of the first relevant document: q1 = 1, q2 = 3, q3 = not found in top 10. Each query has exactly 1 relevant document.

• Recall@5 = 2/3 = 0.667; Recall@1 = 1/3 = 0.333
• MRR = (1/1 + 1/3 + 0)/3 = 1.3333/3 = 0.444
• nDCG@3 for a separate, graded query q4 whose top-3 relevance grades are [3, 0, 2]:

DCG = 3/log₂2 + 0/log₂3 + 2/log₂4 = 3/1 + 0 + 2/2 = 4.000 Ideal order [3, 2, 0]: IDCG = 3/1 + 2/1.585 + 0 = 3 + 1.262 = 4.262 nDCG@3 = 4.000/4.262 = 0.939

Example 5 — storage, latency and cost of a 1-million-chunk index. d = 768, N = 1,000,000.

• float32: 1,000,000 × 768 × 4 = 3,072,000,000 bytes = 3.07 GB (≈2.86 GiB), plus index overhead (an HNSW graph at M = 16–32 adds about 2M × 4 bytes = 128–256 bytes per vector, i.e. only ~5–10% on top of a 768-dim float32 vector; the graph dominates only when vectors are quantised or low-dimensional).
• int8 quantised: 768 bytes/vector → 768 MB; binary (1 bit/dim): 96 bytes/vector → 96 MB.
• Brute-force scan: 1,000,000 × 768 ≈ 7.68 × 10⁸ multiply-adds per query ≈ 1.5 GFLOP — tens of milliseconds on a tuned CPU, which is why ANN is used beyond a few hundred thousand vectors.
• Truncating to d = 256 (Matryoshka) cuts storage to 1.02 GB and compute 3× for a small recall loss.
• One-time embedding: 1,000,000 chunks × 300 tokens = 300 million tokens; at an illustrative ₹2 per million tokens that is about ₹600 — vendor rates change, so price from the current rate card. A self-hosted 384-dim model on one GPU is often cheaper at this volume.

Where it is used

• E-commerce and marketplace search: Flipkart, Myntra, Meesho, BigBasket — matching misspelt, code-mixed queries to catalogue titles; embedding-based "similar products".
• Banking, insurance and fintech support: HDFC, ICICI, Bajaj Finserv, Policybazaar — routing customer queries to the right policy clause or FAQ, deduplicating complaint tickets, matching KYC names.
• IT services and GCC delivery: TCS, Infosys, Wipro, Accenture, Cognizant — enterprise document search, contract and resume–JD matching, log and incident clustering; "Search Relevance Engineer" and "ML Engineer (NLP)" roles.
• Government and public platforms: DigiLocker/UMANG-style FAQ search, judgment and legal search (eCourts, Indian Kanoon-type products), multilingual grievance classification on CPGRAMS-like systems.
• Job and education portals: semantic matching of candidate profiles to vacancies, course and skill recommendations, duplicate-posting detection.
• Content moderation, plagiarism and fraud: near-duplicate detection across languages, clustering of fake reviews, screening repeated insurance claim narratives.

Interview questions

• Q: What is a text embedding and how does it differ from one-hot encoding? A: An embedding is a dense low-dimensional vector in which distance reflects meaning; one-hot vectors are vocabulary-sized, orthogonal, and give every pair of distinct words the same (zero) similarity.
• Q: Why is cosine similarity preferred over Euclidean distance for text? A: Cosine ignores vector magnitude, which varies with document length and word frequency, and compares only direction — the semantic part. After L2 normalisation the two give identical rankings anyway.
• Q: CBOW or skip-gram — which would you use on a small domain corpus? A: Skip-gram with negative sampling, because it learns better representations for rare words, which dominate a small specialised corpus; CBOW is faster and favours frequent words.
• Q: Why does negative sampling exist? A: The full softmax over a vocabulary of hundreds of thousands of words is prohibitively expensive per update; negative sampling approximates it with one positive and k random negatives as binary logistic problems.
• Q: Why are raw BERT CLS embeddings bad for semantic similarity? A: BERT is pretrained on masked-language-modelling, not similarity, so its sentence-level space is anisotropic and poorly aligned; SBERT-style contrastive fine-tuning with a siamese architecture fixes it.
• Q: Bi-encoder vs cross-encoder — when do you use each? A: A bi-encoder embeds query and documents independently so document vectors are precomputed and search is fast; a cross-encoder reads the pair together and is much more accurate, so it reranks only the shortlisted top-k.
• Q: Semantic search returns topically right but factually wrong documents for "loan NOT approved". Why? A: Most encoders encode topic strongly and negation weakly, so the negated and affirmative sentences sit close; fix with negation-aware training data, hybrid lexical matching or a cross-encoder reranker.
• Q: Does a higher embedding dimension always give better retrieval? A: No — quality depends on training data and objective, not size; beyond what the model needs, extra dimensions cost memory and latency, and Matryoshka models often lose very little when truncated.
• Q: How do you choose a similarity threshold for "duplicate"? A: Empirically, by labelling a few hundred pairs and picking the cut-off that meets the required precision/recall on your own data — thresholds do not transfer between models.
• Q: Why does BM25 sometimes beat a good embedding model? A: For rare exact tokens — part numbers, HSN codes, person names, error codes — lexical match is precise while a dense model blurs them into similar-looking neighbours; hence hybrid retrieval.
• Q: You switched the embedding model. What must you do? A: Re-embed the entire corpus and rebuild the index; mixing vectors from two models in one space is meaningless, and the query encoder must be the same model.
• Q: Are stored embeddings anonymous data? A: No — they are derived from personal data, can be partially inverted to recover the source text, and so remain personal data under the Digital Personal Data Protection Act, 2023, requiring the same consent, retention and security controls.

Common mistakes

• Embedding the corpus with one model and the query with another, or forgetting the model's "query:"/"passage:" prefix convention.
• Mixing metrics — scoring with cosine a model trained for inner product, or comparing un-normalised vectors by Euclidean distance.
• Copying a similarity threshold (for example 0.8) from a tutorial instead of calibrating on labelled pairs.
• Ignoring the encoder's maximum sequence length, so long chunks are silently truncated and the tail of the document is never searchable.
• Reporting only Recall@k while hiding latency, cost and the fact that recall was measured on a handful of easy queries.
• Using t-SNE or UMAP coordinates for retrieval or clustering decisions — they preserve local structure only.
• Treating semantic search as an answer engine: it ranks passages by similarity and does nothing about correctness, recency or authority.

Related topics

Natural language processing basics · Retrieval-augmented generation and vector databases · Large language models and prompt engineering · Recommender systems · Information retrieval and ranking · Dimensionality reduction and clustering