Natural language processing basics

In short

Natural Language Processing (NLP) is the branch of AI that lets computers read, interpret and generate human language — text or speech — by combining linguistics, statistics and machine learning. It matters because most enterprise data is unstructured text: support tickets, policy documents, claim notes, court orders, chat logs. Indian university syllabi (AICTE model curriculum for AI&DS, UGC CBCS) examine the classical pipeline — tokenization, morphology, n-gram language models, POS tagging with HMMs, parsing, information extraction and text classification — while GATE DA/CS asks TF-IDF, Naive Bayes, perplexity and edit-distance numericals. Placement rounds at TCS, Infosys, Wipro, Accenture, Fractal, Zoho, Freshworks, Jio Haptik, Gupshup and bank/insurance analytics teams test exactly these fundamentals before any LLM question.

Explained

What NLP is, and why language is hard

• Definition: computational techniques to analyse and generate natural language at the level of characters, words, sentences and discourse.
• NLU vs NLG: Natural Language *Understanding* maps text to meaning (classification, parsing, extraction); Natural Language *Generation* maps meaning to text (summaries, replies).
• Levels of analysis (standard exam list): phonology (sound) → morphology (word structure) → lexical (words and their senses) → syntax (grammar) → semantics (meaning) → pragmatics (intent in context) → discourse (across sentences).
• Why hard — ambiguity at every level: lexical ("bank" = river bank / financial bank), structural/syntactic ("I saw the man with the telescope"), referential ("Ravi told Arun he had failed" — who?), semantic, and pragmatic ("Can you pass the salt?" is a request, not a question).
• Other difficulties: idioms, sarcasm, code-mixing (Hinglish), spelling variation in transliteration (Krishna/Krishnaa/Krsna), rich morphology and agglutination in Tamil, Telugu, Kannada, Malayalam; free word order in Hindi; no capitalisation cue in Indic scripts; sandhi (word joining) in Sanskrit-derived languages.
• Indian scripts: encoded in Unicode (Devanagari U+0900–097F etc.); legacy ISCII is largely obsolete. Government-funded resources come from MeitY's Bhashini / National Language Translation Mission, TDIL (Technology Development for Indian Languages), AI4Bharat (IIT Madras — IndicNLP corpora, IndicBERT, IndicTrans) and C-DAC.

Text preprocessing pipeline

1. Text acquisition and cleaning: strip HTML, fix encoding, Unicode normalisation (NFC/NFKC) so visually identical Devanagari sequences compare equal.
2. Sentence segmentation: split on ., ?, !, and the Devanagari danda ।; abbreviations ("Dr.", "Rs.") and decimals break naive splitting.
3. Tokenization: split into tokens (words, punctuation). Types — whitespace, rule-based/regex, Treebank, and subword (BPE, WordPiece, SentencePiece) which handles out-of-vocabulary words and suits morphologically rich Indian languages.
4. Case folding: lowercasing helps matching but destroys cues (US vs us); irrelevant for Indic scripts, which are unicase.
5. Stop-word removal: drop high-frequency function words (the, is, of; का, है, और). Harmful for sentiment ("not good") and phrase search.
6. Stemming: crude suffix chopping to a stem; Porter stemmer (five rule phases), Lovins, Snowball. Fast, language-specific, may produce non-words ("studies" → "studi").
7. Lemmatization: dictionary- plus POS-based reduction to the valid dictionary form (lemma): "better" → "good", "running/ran" → "run". Slower, more accurate, needs a lexicon such as WordNet (IndoWordNet covers 18 Indian languages, from IIT Bombay).
8. Other steps: spell correction, normalisation of numbers/dates/currency (₹1,00,000 → NUM), de-accenting, removing rare terms, handling emojis and URLs.
9. Morphology terms: morpheme (smallest meaning unit), root/stem, affix (prefix/suffix/infix), inflectional morphology (plays, played — same word class) vs derivational (play → playful — new class). Finite-state transducers (FSTs) are the classical morphological analyser; the two-level/Kimmo model is a syllabus favourite.

Representing text: from counts to vectors

• Vocabulary V, document-term matrix: rows = documents, columns = terms.
• One-hot encoding: one dimension per word; sparse, equidistant, no notion of similarity.
• Bag of Words (BoW): counts of words, order discarded. Simple, strong baseline for classification.
• n-grams: contiguous sequences of n tokens; bigrams/trigrams restore limited word order ("not good" survives as a bigram). Character n-grams help language identification and noisy text.
• TF-IDF: weights a term high if frequent in a document but rare across the corpus; damps common words without a stop list.
• Cosine similarity compares documents regardless of length; Jaccard compares term sets.
• Limitations of count vectors: very high dimensionality, sparsity, no synonymy ("car"/"vehicle" unrelated), no polysemy handling. Reduced by feature selection (chi-square, mutual information) or dimensionality reduction (LSA/SVD, LDA topic modelling).
• Dense distributed representations (word2vec, GloVe, contextual embeddings) are the successor family — covered as its own topic; here it is enough to know *why* counts fail.

Language models and probability

• Task: assign a probability to a word sequence; used in spell/grammar correction, speech recognition, machine translation, input-method prediction (Google Indic keyboard).
• Chain rule + Markov assumption: an n-gram model conditions each word only on the previous n−1 words — unigram, bigram, trigram.
• Estimation: maximum likelihood = relative frequency of counts.
• Zero-probability problem and smoothing: Laplace (add-one), add-k, Good-Turing, backoff (Katz), interpolation (Jelinek-Mercer), Kneser-Ney (best classical smoother, uses continuation counts).
• Evaluation: extrinsic (end-task score) vs intrinsic — perplexity, the inverse geometric mean probability per word; lower is better. Perplexity is comparable only across models with the same vocabulary and test set.
• Neural LMs: feed-forward → RNN/LSTM (handles longer context, suffers vanishing gradients) → Transformer with self-attention (parallel training, long-range dependencies), the basis of modern pretrained models.

Core sequence and structure tasks

• POS tagging: assign noun/verb/adjective tags; tagsets — Penn Treebank (45 tags), Universal Dependencies (17), and the BIS tagset standard for Indian languages. Approaches: rule-based, stochastic (HMM + Viterbi decoding), transformation-based (Brill), neural (BiLSTM-CRF, BERT).
• HMM essentials: states = tags, observations = words, parameters = transition P(tag|prev tag) and emission P(word|tag); three problems — evaluation (forward algorithm), decoding (Viterbi), learning (Baum-Welch).
• Chunking (shallow parsing): find non-overlapping NP/VP chunks using BIO/IOB tagging (B-NP, I-NP, O).
• Named Entity Recognition (NER): label spans as PERSON, ORG, LOC, DATE, MONEY; also BIO-tagged. Indian text is hard — person names double as common nouns (Kamal, Akash) and there is no capitalisation signal in Indic scripts.
• Parsing: constituency parsing with a context-free grammar (CFG: terminals, non-terminals, start symbol, production rules) using CYK (bottom-up, needs Chomsky Normal Form, O(n³·|G|)) or Earley (handles arbitrary CFGs); PCFG adds rule probabilities to pick the most likely tree. Dependency parsing gives head–dependent relations (nsubj, obj) — preferred for free-word-order Indian languages; transition-based (shift-reduce, arc-standard) or graph-based (MST).
• Ambiguity in parsing: attachment ambiguity and coordination ambiguity cause multiple valid trees; PCFG probabilities or lexicalisation resolve them.
• Semantics: word senses, WSD (Lesk algorithm — overlap of gloss words; supervised classifiers), semantic roles (SRL: agent, patient, instrument), lexical relations (synonym, antonym, hypernym/hyponym, meronym), coreference resolution, first-order-logic and lambda-calculus meaning representations.
• Information extraction: relation extraction, event extraction, template filling; distant supervision.

Applied tasks, evaluation and tooling

• Text classification: spam detection, sentiment analysis, intent detection, topic labelling. Classical models — Multinomial Naive Bayes (fast, assumes conditional independence of words), logistic regression, linear SVM (strong on sparse TF-IDF); then fine-tuned transformers.
• Sentiment analysis: lexicon-based (SentiWordNet, VADER) vs supervised; aspect-based sentiment separates "battery good, camera poor"; negation scope and sarcasm are the known failure points.
• Other tasks: machine translation (rule-based → SMT with IBM models and BLEU → neural MT), summarization (extractive vs abstractive), question answering, dialogue systems, text normalisation, transliteration (ITRANS, WX notation for Indian languages).
• Evaluation metrics: accuracy; precision, recall, F1 (macro vs micro averaging) for imbalanced classes; confusion matrix; for sequence labelling, entity-level exact-match F1; BLEU (MT, n-gram precision + brevity penalty), ROUGE (summarization, recall of n-grams/LCS), METEOR, WER (speech); perplexity for LMs. Always report with train/validation/test splits and k-fold cross-validation.
• Tools and corpora: Python with NLTK (teaching), spaCy (production pipelines), scikit-learn (vectorizers and classifiers), Gensim (topic models), Stanford CoreNLP, Hugging Face Transformers; Indic libraries — iNLTK, IndicNLP Library, AI4Bharat IndicTrans/IndicBERT; corpora — Brown, Penn Treebank, IIT Bombay English-Hindi corpus, Samanantar, IndicCorp, IndicGLUE benchmark.
• Legal and ethical context in India: the Digital Personal Data Protection Act, 2023 governs processing of personal data in text corpora (notice, consent, purpose limitation, data-principal rights); rules operationalising it have been notified in stages, so check the current position before designing a pipeline. Use the Information Technology Act, 2000 (with the 2011 sensitive-personal-data rules and the 2021 intermediary guidelines) for content and intermediary duties; RBI's data-localisation direction applies to payment-related text data. Bias, PII leakage, consent for scraped data and de-identification (masking Aadhaar/PAN/phone patterns by regex) are standard viva questions.

Key points

• NLP - computational analysis and generation of human language; NLU (text→meaning) and NLG (meaning→text).
• Levels - phonology, morphology, lexical, syntax, semantics, pragmatics, discourse.
• Ambiguity - lexical, syntactic, referential, semantic, pragmatic; the central difficulty of NLP.
• Tokenization - splitting text into tokens; subword tokenizers (BPE/WordPiece/SentencePiece) solve OOV words.
• Type vs token - distinct words vs total word occurrences; vocabulary size = number of types.
• Stemming vs lemmatization - rule-based chopping to a possibly invalid stem vs dictionary+POS reduction to a valid lemma.
• Bag of Words - word counts, order discarded; n-gram restores local order.
• TF-IDF - term frequency × inverse document frequency; high for terms frequent here and rare elsewhere.
• Cosine similarity - length-independent document similarity, range 0 to 1 for non-negative vectors.
• Zipf's law - word frequency is roughly inversely proportional to frequency rank; explains sparsity.
• n-gram LM - Markov approximation of P(sequence); needs smoothing (Laplace, Good-Turing, Kneser-Ney) for unseen n-grams.
• Perplexity - intrinsic LM metric = 2^{cross-entropy}; lower is better; comparable only on the same vocabulary and test set.
• HMM POS tagging - states are tags, emissions are words; Viterbi finds the most likely tag sequence in O(n·|T|²).
• BIO tagging - encodes spans for chunking and NER (B = begin, I = inside, O = outside).
• CFG parsing - CYK is bottom-up and needs CNF, O(n³); PCFG picks the most probable parse tree.
• Dependency parsing - head-dependent relations; better for free-word-order Indian languages than constituency parsing.
• Naive Bayes - fast text classifier assuming word independence; needs Laplace smoothing for unseen words.
• Metrics - F1 for imbalanced classification, BLEU for translation, ROUGE for summarization, WER for speech.

Formulas and examples

Example 1 — TF-IDF by hand. Corpus of N = 4 support tickets. The term "refund" appears in 2 of them. In ticket d₁ of 50 tokens, "refund" appears 3 times.

• tf = 3/50 = 0.06
• idf = log₁₀(4/2) = log₁₀ 2 = 0.3010
• w = 0.06 × 0.3010 = 0.01806
• The word "the" appears in all 4 tickets → idf = log₁₀(4/4) = 0, so its TF-IDF weight is 0 whatever its count. This is why TF-IDF needs no stop-word list.

Example 2 — Cosine similarity of two documents. Vocabulary (loan, EMI, delay, refund). A = (2, 1, 0, 0), B = (3, 0, 1, 0).

• Dot product = 2×3 + 1×0 + 0×1 + 0×0 = 6
• |A| = √(4+1) = √5 = 2.2361; |B| = √(9+0+1) = √10 = 3.1623
• cos θ = 6 / (2.2361 × 3.1623) = 6 / 7.0711 = 0.8485 (≈ 32° apart — strongly similar)

Example 3 — Bigram probability and perplexity. Training corpus (3 sentences, <s> and </s> added): <s> I like NLP </s>, <s> I like tea </s>, <s> I love NLP </s> Counts: C(<s>) = 3, C(<s> I) = 3, C(I) = 3, C(I like) = 2, C(I love) = 1, C(like) = 2, C(like NLP) = 1, C(like tea) = 1, C(NLP) = 2, C(NLP </s>) = 2.

• MLE: P(I|<s>) = 3/3 = 1; P(like|I) = 2/3 = 0.6667; P(NLP|like) = 1/2 = 0.5; P(</s>|NLP) = 2/2 = 1
• P(<s> I like NLP </s>) = 1 × 0.6667 × 0.5 × 1 = 0.3333
• Perplexity over the N = 4 predicted tokens: PP = 0.3333^−1/4 = (3)^0.25 = 1.316
• Unseen bigram "love tea": MLE gives 0 → whole sentence probability 0. With Laplace and V = 6 types (I, like, love, NLP, tea, </s>): P(tea|love) = (0+1)/(1+6) = 1/7 = 0.1429, no longer zero.

Example 4 — Naive Bayes sentiment classification. Training: 3 positive and 2 negative reviews. Positive class has 10 word tokens with counts {good:4, fast:3, service:3}; negative class has 8 tokens with counts {bad:4, slow:2, service:2}. Vocabulary V = {good, fast, service, bad, slow} → |V| = 5. Classify "good service".

• Priors: P(pos) = 3/5 = 0.6, P(neg) = 2/5 = 0.4
• Laplace, positive: P(good|pos) = (4+1)/(10+5) = 5/15 = 0.3333; P(service|pos) = (3+1)/15 = 0.2667
• Laplace, negative: P(good|neg) = (0+1)/(8+5) = 1/13 = 0.0769; P(service|neg) = (2+1)/13 = 0.2308
• Score(pos) = 0.6 × 0.3333 × 0.2667 = 0.05333
• Score(neg) = 0.4 × 0.0769 × 0.2308 = 0.00710
• Positive score is larger → predict positive (normalised posterior ≈ 0.882).

Example 5 — Edit distance and NER evaluation. (a) Levenshtein distance between "kitten" and "sitting" (unit substitution cost) = 3: k→s, e→i, insert g. (b) An NER model on 100 Hindi news sentences predicts 120 PERSON spans; 90 are exactly correct; the gold data has 150 PERSON spans.

• Precision = 90/120 = 0.75; Recall = 90/150 = 0.60
• F1 = 2 × 0.75 × 0.60 / (0.75 + 0.60) = 0.90/1.35 = 0.6667
• Accuracy is meaningless here because most tokens are "O"; token accuracy could exceed 95% with the same poor F1.

# Classical pipeline: TF-IDF + Naive Bayes on support tickets
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.metrics import classification_report

X_train = ["refund not received", "EMI debited twice", "loan approved fast"]
y_train = ["refund", "payment", "loan"]

clf = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, min_df=1),
    MultinomialNB(alpha=1.0),          # alpha = Laplace smoothing
)
clf.fit(X_train, y_train)
print(clf.predict(["my refund is pending"]))   # -> ['refund']

Where it is used

• IT services and BPM (TCS, Infosys, Wipro, Cognizant, Accenture): ticket triage and auto-routing, contract and invoice extraction, email classification, RPA document pipelines — the largest NLP employer base in India.
• Banking, insurance and fintech (HDFC, ICICI, SBI, Bajaj Finserv, Razorpay, PolicyBazaar): KYC document parsing, claims-note extraction, complaint categorisation for RBI/IRDAI reporting, AML adverse-media screening, chatbots like SBI's SIA.
• Conversational AI and CX products (Jio Haptik, Gupshup, Yellow.ai, Freshworks, Zoho, Ozonetel): intent classification, entity slot filling, multilingual and code-mixed Hinglish bots for WhatsApp support.
• E-commerce and consumer internet (Flipkart, Amazon India, Swiggy, Zomato, Myntra): review sentiment and aspect mining, search query understanding, catalogue attribute extraction, abuse and spam moderation.
• Government and public interest: Bhashini/NLTM translation and voice services for citizen apps, e-Courts judgment search and headnote generation, grievance classification on CPGRAMS, Census/survey open-response coding.
• Job roles: NLP engineer, data scientist, computational linguist, annotation/linguistic QA lead, ML engineer (MLOps for text), research intern at AI4Bharat, IIIT-Hyderabad LTRC or Microsoft Research India.

Interview questions

• Q: What is the difference between stemming and lemmatization, and when would you prefer stemming? A: Stemming chops suffixes by rules and may give a non-word; lemmatization uses a dictionary and POS to return a valid lemma. Stemming is preferred when speed matters and the output is only an index key, as in large-scale search.
• Q: Why does TF-IDF outperform raw counts? A: IDF down-weights terms that appear in most documents and therefore carry no discriminative power, while keeping terms that are frequent in a document but rare in the corpus.
• Q: What problem does smoothing solve in an n-gram language model? A: Unseen n-grams get zero MLE probability, which makes the whole sequence probability zero; smoothing shifts a little probability mass to unseen events so the model can score any sentence.
• Q: What does perplexity of 120 mean? A: The model is, on average, as uncertain as if choosing uniformly among 120 words at each position; it is only comparable against another model on the same test set and vocabulary.
• Q: Why is accuracy a bad metric for NER or spam detection? A: Classes are highly imbalanced — most tokens are "O" and most mail is ham — so a trivial majority predictor scores high accuracy; precision, recall and F1 on the positive class are needed.
• Q: How does the Viterbi algorithm tag a sentence? A: It does dynamic programming over tag states, keeping for each word the best-scoring path using transition and emission probabilities, then back-traces the maximum-probability tag sequence in O(n·|T|²).
• Q: Why is Naive Bayes called "naive" yet works well on text? A: It assumes words are conditionally independent given the class, which is false; but the decision only needs the correct argmax, and sparse high-dimensional text data makes that ranking robust.
• Q: Should stop words always be removed? A: No. They hurt tasks where function words carry meaning — sentiment with negation ("not good"), authorship attribution, phrase search and parsing; TF-IDF or a model can down-weight them instead.
• Q: Why do subword tokenizers help with Indian languages? A: Tamil, Telugu, Kannada and Malayalam are morphologically rich and agglutinative, so word-level vocabularies explode and hit many out-of-vocabulary forms; BPE/SentencePiece pieces compose unseen words from known fragments.
• Q: Constituency or dependency parsing for Hindi — which and why? A: Dependency parsing, because Hindi has relatively free word order and case markers; head-dependent relations capture the predicate-argument structure without committing to a fixed phrase order.
• Q: How would you handle code-mixed Hinglish user messages? A: Normalise and transliterate to a single script, use character/subword features or a multilingual pretrained model (IndicBERT, MuRIL), and train on code-mixed annotated data rather than pure-Hindi or pure-English corpora.
• Q: What legal constraint applies when you scrape Indian customer reviews containing names and phone numbers? A: Personal data processing falls under the Digital Personal Data Protection Act, 2023 (notice, consent, purpose limitation, data-principal rights), so the pipeline should mask or hash PII and retain only what the stated purpose needs; the implementing rules are being notified in stages, so verify the current requirements.

Common mistakes

• Fitting the vectorizer on the full dataset before splitting — the IDF statistics leak test information and inflate scores; fit on train, transform test.
• Removing stop words or punctuation blindly, then wondering why negation and sarcasm handling collapses in sentiment tasks.
• Computing IDF with the number of *occurrences* instead of document frequency, or forgetting which log base is used (the base only scales all weights equally, but must be stated).
• Reporting accuracy on imbalanced text data, and reporting token-level scores for NER where entity-level exact-match F1 is required.
• Forgetting to add V (vocabulary size) to the denominator in Laplace smoothing, or counting the same type twice when fixing V.
• Comparing perplexity across models with different vocabularies, tokenizers or test sets, and treating lower perplexity as proof of better end-task performance.
• Using N instead of N+1 token positions when sentence boundary markers <s>/</s> are added, so bigram counts and perplexity denominators go wrong.
• Assuming an English pipeline transfers to Indic text — skipping Unicode normalisation, splitting sentences only on "." and ignoring the danda ।.

Related topics

Text embeddings and semantic search · Large language models and prompt engineering · Speech recognition and audio AI · Machine learning classification and evaluation metrics · Probability and information theory (entropy, Bayes) · Automata and formal languages (regular expressions, CFGs)