Data mining and knowledge discovery

In short

Data mining is the extraction of previously unknown, valid and useful patterns from large volumes of data; knowledge discovery in databases (KDD) is the wider nine-step process - selection, preprocessing, transformation, mining, evaluation and presentation - in which mining is only one step. It answers questions a query cannot: which items are bought together, which customer will churn, which transaction is fraudulent, which patients group together. MCA syllabi examine it as a full paper (KDD steps, Apriori with support/confidence tables, decision trees with entropy and information gain, k-means iterations, Bayes classification), and GATE/UGC-NET ask the same numerics. Indian employers use it directly - credit scoring at HDFC and Bajaj Finserv, recommendation at Flipkart and Myntra, fraud analytics on UPI at NPCI, churn models at Jio and Airtel, and UIDAI/GSTN analytics - with personal-data handling now governed by the DPDP Act, 2023.

Explained

What data mining is, and what it is not

• Definition: the non-trivial extraction of implicit, previously unknown and potentially useful information from data (Frawley, Piatetsky-Shapiro). "Non-trivial" excludes a simple SQL aggregate.
• Data mining is one step of KDD, although industry uses the two names interchangeably. The exam answer: mining = pattern extraction; KDD = the whole pipeline ending in actionable knowledge.
• Not a query language, not reporting, not OLAP. OLAP is user-driven and verification-based (you ask, it slices); mining is data-driven and discovery-based (it proposes the hypothesis).
• Interestingness of a pattern: must be *valid* on new data, *novel*, *useful* and *understandable*. Measured objectively (support, confidence, lift, accuracy) and subjectively (unexpectedness, actionability).
• Data sources mined: relational databases, data warehouses, transactional databases, object/temporal/spatial data, time-series and streams, text, web logs, graphs and social networks.
• Two broad task families:
  • Descriptive (unsupervised) - characterise what is in the data: clustering, association rules, summarisation, outlier detection.
  • Predictive (supervised) - predict an unknown value: classification (categorical target), regression (numeric target).

The KDD process, step by step

1. Domain understanding and goal setting - state the business question (reduce churn by 5%).
2. Data selection - choose target data set, tables, attributes, rows, time window.
3. Data cleaning - handle missing values (ignore tuple, fill mean/median/mode, predict), smooth noise (binning, regression), remove duplicates, correct inconsistencies.
4. Data integration - merge multiple sources; resolve schema conflicts, entity identification (cust_id vs customer_no) and redundancy (correlation / chi-square test).
5. Data transformation - normalisation, aggregation, generalisation (concept hierarchy: city → state → zone), attribute construction (ratio of credit used), discretisation (binning age into bands).
6. Data reduction - dimensionality reduction (PCA, attribute subset selection), numerosity reduction (sampling, clustering, histograms), data-cube aggregation, compression.
7. Data mining - choose the task and algorithm, fit the model.
8. Pattern evaluation - keep only interesting patterns using thresholds and measures.
9. Knowledge presentation - visualisation, rules, dashboards, deployment into a scoring system.
10. Rule of thumb: steps 2-6 (preprocessing) consume roughly 60-80% of project effort. Garbage in, garbage out.
11. CRISP-DM is the industry counterpart: business understanding → data understanding → data preparation → modelling → evaluation → deployment, drawn as a cycle. SEMMA (SAS) = Sample, Explore, Modify, Model, Assess.

Data preprocessing in detail

• Data quality dimensions: accuracy, completeness, consistency, timeliness, believability, interpretability.
• Missing data: MCAR/MAR/MNAR distinction matters - mean imputation on data missing *because* of the value biases the model.
• Noise smoothing: equal-width binning (equal ranges) vs equal-frequency/equal-depth binning (equal counts); smoothing by bin mean, median or boundary.
• Normalisation methods: min-max (fixed range), z-score (mean 0, SD 1), decimal scaling. Essential for distance-based methods (k-NN, k-means, SVM, neural nets); irrelevant for decision trees and Naive Bayes.
• Encoding: label encoding (ordinal) vs one-hot encoding (nominal). Label-encoding a nominal attribute invents a false ordering.
• Imbalanced data: oversampling, SMOTE (synthetic minority samples), undersampling, class weights. A 1:1000 fraud data set gives 99.9% accuracy to a model that predicts "not fraud" always.
• Train/validation/test split and k-fold cross-validation; stratified folds keep class proportions. The test set must never influence preprocessing (fit the scaler on train only - leakage otherwise).

Association rule mining

• Terms: *item*, *itemset*, *transaction database* D of |D| transactions. A rule is X ⇒ Y with X ∩ Y = ∅.
• Support(X) = fraction of transactions containing X. Frequent itemset = support ≥ min_sup.
• Confidence(X ⇒ Y) = support(X ∪ Y)/support(X) - conditional probability P(Y|X). Strong rule = support ≥ min_sup and confidence ≥ min_conf.
• Lift = confidence/support(Y). Lift > 1 positive correlation, = 1 independent, < 1 negative. Confidence alone is misleading when Y is very frequent.
• Apriori property (downward closure): every non-empty subset of a frequent itemset is frequent; equivalently, if an itemset is infrequent, all its supersets are infrequent - this prunes the search.
• Apriori algorithm: (a) scan D, find frequent 1-itemsets L1; (b) join L(k-1) with itself on the first k-2 items to form candidates Ck; (c) prune candidates having any infrequent (k-1)-subset; (d) scan D to count support, keep Lk; (e) repeat until Lk is empty; (f) generate rules from each frequent itemset by splitting it into antecedent/consequent and testing confidence.
• Weaknesses: multiple database scans (k+1 scans), huge candidate set when min_sup is low.
• FP-growth: builds a compact FP-tree in two scans (first for item frequencies, second to insert sorted transactions along shared prefixes), then mines recursively by conditional pattern bases - no candidate generation; usually an order of magnitude faster.
• Other variants: ECLAT (vertical TID-list format, support by TID-set intersection), partitioning and sampling methods, multilevel rules (concept hierarchies), multidimensional rules, quantitative rules, closed and maximal frequent itemsets (compressed representations), correlation rules using lift/chi-square/all-confidence.
• Market-basket meaning: "bread ⇒ butter" with support 0.4, confidence 0.8 means 40% of bills contain both and 80% of bread buyers also buy butter.

Classification and prediction

• Two phases: learning (build model from labelled training set) and classification (apply to unseen tuples, after measuring accuracy on a test set).
• Decision tree induction (ID3, C4.5, CART): greedy, top-down, recursive partitioning; choose the attribute with the best split measure; stop when all tuples belong to one class, no attributes remain, or no tuples remain (majority vote).
  • ID3 uses information gain (biased towards many-valued attributes), handles only categorical attributes.
  • C4.5 uses gain ratio (gain normalised by split information), handles continuous attributes, missing values, and does pessimistic pruning; it produces multiway splits.
  • CART uses Gini index, gives strictly binary splits, supports regression trees, uses cost-complexity pruning.
  • Pruning: pre-pruning (stop early on a threshold) vs post-pruning (grow fully then cut back); fights overfitting.
  • Advantages: interpretable, no normalisation needed, handles mixed attributes. Limitations: unstable (small data change → different tree), axis-parallel boundaries, can overfit.
• Naive Bayes: applies Bayes' theorem with the class-conditional independence assumption; needs only counts; very fast; Laplace (add-one) smoothing avoids a zero probability wiping out the product. Works well for text/spam despite the unrealistic assumption.
• k-Nearest Neighbour: lazy/instance-based learner, no training phase; classify by majority class among k nearest tuples (Euclidean distance after normalisation). Choice of k: small k = noisy, large k = over-smoothed; odd k avoids ties. Costly at prediction time.
• Rule-based classifiers: IF-THEN rules, extracted from trees or by sequential covering (RIPPER); rule coverage and accuracy.
• Other classifiers: backpropagation neural networks, support vector machines (maximum-margin hyperplane, kernel trick), ensembles - bagging (parallel, variance reduction, random forest adds random feature subsets), boosting (sequential, reweights errors - AdaBoost, gradient boosting/XGBoost), stacking.
• Linear and logistic regression for numeric prediction and probability of a class respectively.
• Evaluation: confusion matrix, accuracy, error rate, precision, recall (sensitivity), specificity, F1, ROC curve and AUC, Matthews correlation. Use holdout, random subsampling, k-fold cross-validation, bootstrap (.632).
• Bias-variance: underfitting = high bias (too simple); overfitting = high variance (memorises training noise). Remedies: more data, fewer features, pruning, regularisation, cross-validation.

Clustering and outlier detection

• Clustering = unsupervised grouping maximising intra-cluster similarity and minimising inter-cluster similarity. Requirements: scalability, arbitrary shapes, minimal domain input, noise tolerance, high dimensionality, interpretability.
• Distance measures: Euclidean, Manhattan, Minkowski (general, with parameter p), cosine similarity (text), Jaccard (binary sets), Hamming.
• Partitioning - k-means: choose k, initialise centroids, assign each point to nearest centroid, recompute centroids as means, repeat until no reassignment. Minimises SSE (within-cluster sum of squares). Fast - O(n k t d) - but needs k in advance, assumes spherical equal-size clusters, sensitive to outliers and to initial seeds (k-means++ fixes seeding). k-medoids (PAM) uses actual objects as centres, robust to outliers, slower.
• Choosing k: elbow method on SSE, silhouette coefficient, gap statistic.
• Hierarchical: agglomerative (AGNES, bottom-up merging) and divisive (DIANA, top-down); linkage = single (chaining, elongated clusters), complete (compact), average, centroid, Ward's. Output is a dendrogram cut at a chosen height. No k needed, but O(n²-n³) and merges are irreversible. BIRCH scales it with a CF-tree.
• Density-based - DBSCAN: parameters ε and MinPts; classifies points as core, border or noise; finds arbitrary shapes and labels outliers; struggles with varying density and high dimensions. OPTICS orders points to handle varying density.
• Grid-based: STING, CLIQUE (also a subspace clustering method). Model-based: EM with Gaussian mixtures, SOM.
• Outlier/anomaly detection: statistical (z-score, IQR, Grubbs), distance-based, density-based LOF, clustering-based, isolation forest, one-class SVM. Used for fraud, intrusion detection, quality control. An outlier may be noise to delete *or* the very pattern wanted - decide from the domain.

Advanced mining, limitations and the Indian legal frame

• Text mining: tokenisation, stop-word removal, stemming/lemmatisation, bag-of-words, TF-IDF weighting, vector space model, cosine similarity, topic models (LDA), sentiment analysis.
• Web mining: web content mining, web structure mining (hyperlink graph, PageRank, HITS) and web usage mining (clickstream/log analysis, sessionisation).
• Sequence and time-series mining: sequential patterns (GSP, PrefixSpan), trend and periodicity analysis, motif discovery.
• Graph mining (frequent subgraphs, community detection), spatial mining (GIS), stream mining (one-pass, limited memory - Hoeffding trees, lossy counting, concept drift).
• Limitations and risks: correlation is not causation; spurious patterns multiply when you test millions of rules; data quality bottleneck; scalability and high-dimensional "curse of dimensionality"; models decay as behaviour drifts; discrimination and privacy harm; black-box models unacceptable in regulated lending.
• Privacy-preserving data mining: anonymisation (k-anonymity, l-diversity, t-closeness), perturbation/randomisation, secure multiparty computation, differential privacy, federated learning.
• Indian law and practice (as of November 2026):
  • The Digital Personal Data Protection Act, 2023 governs digital personal data - consent notice, purpose limitation, data-minimisation, rights of access/correction/erasure, significant data fiduciary duties (DPIA, audit) and the Data Protection Board. Under the DPDP Rules, 2025 the main Data Fiduciary duties (notice, consent, breach intimation, erasure) and the omission of IT Act section 43A apply only from 13 May 2027; until then IT Act section 43A and the SPDI Rules, 2011 apply to sensitive personal data. Verify current status before relying on it in a compliance answer.
  • RBI's master directions on credit models and digital lending require explainable, auditable loan decisioning and data localisation of payment data (RBI's 2018 storage mandate); credit-information use is governed by the CIC (Regulation) Act, 2005.
  • Anonymised/non-personal data falls outside the DPDP Act, which is why aggregated GSTN/UIDAI-style analytics are usually structured on anonymised sets.

Key points

• Data mining - non-trivial discovery of valid, novel, useful, understandable patterns in large data.
• KDD - the nine-step pipeline (selection → preprocessing → transformation → mining → evaluation → presentation); mining is one step.
• OLAP vs mining - OLAP verifies a hypothesis you supply; mining generates the hypothesis.
• Descriptive vs predictive - clustering/association/outliers vs classification/regression.
• Support - how often an itemset occurs; confidence - conditional probability of the consequent; lift - independence check.
• Apriori property - all subsets of a frequent itemset are frequent; its contrapositive does the pruning.
• FP-growth - two scans, FP-tree, no candidate generation; faster than Apriori at low min_sup.
• Information gain - entropy reduction from a split (ID3); gain ratio (C4.5) corrects its bias towards many-valued attributes; Gini (CART) gives binary splits.
• Naive Bayes - class-conditional independence; needs Laplace smoothing; strong baseline for text.
• k-NN - lazy learner, no model, needs normalisation and a good k.
• k-means - minimises SSE, needs k, spherical clusters, outlier-sensitive; k-medoids robust, DBSCAN finds arbitrary shapes and noise.
• Hierarchical clustering - dendrogram, no k needed, O(n²) or worse, merges irreversible.
• Confusion matrix - TP/FP/FN/TN; accuracy is useless on imbalanced data - quote precision, recall, F1, AUC.
• Overfitting - excellent on training data, poor on test; cured by pruning, regularisation, more data, cross-validation.
• Curse of dimensionality - distances become uniform in high dimensions; reduce with PCA or feature selection.
• Preprocessing is 60-80% of the work; normalisation matters for distance-based learners only.
• Privacy - k-anonymity, l-diversity, differential privacy; in India the DPDP Act, 2023 with Rules phasing duties in from 13 May 2027 (verify).
• Correlation is not causation - a strong rule is a signal for an experiment, not proof.

Formulas and examples

Example 1 - Apriori on 5 transactions (min_sup = 40%, min_conf = 70%)

• |D| = 5, so min_sup = 40% means a count of at least 2.
• L1 counts: Bread 4 (80%), Milk 4 (80%), Butter 3 (60%), Eggs 3 (60%) - all frequent.
• C2 counts: {Bread,Milk} 3 (T1,T4,T5) = 60%; {Bread,Butter} 2 (T2,T4) = 40%; {Bread,Eggs} 2 (T2,T5) = 40%; {Milk,Butter} 2 (T3,T4) = 40%; {Milk,Eggs} 2 (T3,T5) = 40%; {Butter,Eggs} 2 (T2,T3) = 40%. All six are frequent.
• C3: {Bread,Milk,Butter} 1 (T4); {Bread,Milk,Eggs} 1 (T5); {Bread,Butter,Eggs} 1 (T2); {Milk,Butter,Eggs} 1 (T3) - all count 1 < 2, so L3 = ∅ and the algorithm stops.
• Rules from {Bread, Milk} (support 60%): Bread ⇒ Milk, confidence = 3/4 = 75% (strong); Milk ⇒ Bread, confidence = 3/4 = 75% (strong).
• Lift(Bread ⇒ Milk) = 0.75/0.80 = 0.9375 - below 1, so despite 75% confidence the two are slightly *negatively* associated. This is the classic trap: a strong rule can still be uninteresting.
• Rules from {Butter, Eggs} (support 40%): Butter ⇒ Eggs = 2/3 = 66.7% (rejected, < 70%); Eggs ⇒ Butter = 2/3 = 66.7% (rejected).

Example 2 - Decision tree split by information gain (14 tuples)

Class distribution: 9 "Buy = yes", 5 "Buy = no".

• Info(D) = −(9/14)log2(9/14) − (5/14)log2(5/14) = −0.6429(−0.6374) − 0.3571(−1.4854) = 0.4097 + 0.5305 = 0.9402 bits.
• Split on Age into three branches: youth (5: 2 yes, 3 no), middle (4: 4 yes, 0 no), senior (5: 3 yes, 2 no).
  • Info(youth) = −(2/5)log2(2/5) − (3/5)log2(3/5) = 0.4×1.3219 + 0.6×0.7370 = 0.5288 + 0.4422 = 0.9710.
  • Info(middle) = 0 (pure node).
  • Info(senior) = same shape as youth = 0.9710.
  • Info_Age(D) = (5/14)(0.9710) + (4/14)(0) + (5/14)(0.9710) = 0.3468 + 0 + 0.3468 = 0.6936.
  • Gain(Age) = 0.9402 − 0.6936 = 0.2466 bits.
• Split on Income into high (4: 2 yes, 2 no), medium (6: 4 yes, 2 no), low (4: 3 yes, 1 no).
  • Info(high) = 1.0; Info(medium) = −(4/6)log2(4/6) − (2/6)log2(2/6) = 0.6667×0.5850 + 0.3333×1.5850 = 0.3900 + 0.5283 = 0.9183; Info(low) = −0.75log2 0.75 − 0.25log2 0.25 = 0.75×0.4150 + 0.25×2 = 0.3113 + 0.5 = 0.8113.
  • Info_Income(D) = (4/14)(1.0) + (6/14)(0.9183) + (4/14)(0.8113) = 0.2857 + 0.3936 + 0.2318 = 0.9111.
  • Gain(Income) = 0.9402 − 0.9111 = 0.0291 bits.
• Age wins (0.2466 > 0.0291), so Age becomes the root; the middle-aged branch is already a pure leaf.
• Gain ratio check for Age: SplitInfo = −(5/14)log2(5/14) − (4/14)log2(4/14) − (5/14)log2(5/14) = 0.3571×1.4854 + 0.2857×1.8074 + 0.3571×1.4854 = 0.5305 + 0.5164 + 0.5305 = 1.5774 → GainRatio = 0.2466/1.5774 = 0.156.
• Gini for the same root split: Gini(D) = 1 − (9/14)² − (5/14)² = 1 − 0.4133 − 0.1276 = 0.4592; Gini(youth) = 1 − 0.16 − 0.36 = 0.48, Gini(middle) = 0, Gini(senior) = 0.48 → Gini_Age = (5/14)(0.48) + 0 + (5/14)(0.48) = 0.1714 + 0.1714 = 0.3429 → ΔGini = 0.1163.

Example 3 - k-means, one full iteration (1-D data)

Points: 2, 4, 10, 12, 3, 20, 30, 11, 25. k = 2, initial centroids c1 = 2, c2 = 4.

• Iteration 1 assignment (nearest centroid): c1 → {2}; c2 → {4, 10, 12, 3, 20, 30, 11, 25}. (3 is distance 1 from 2 and 1 from 4 - tie broken to c2.)
• New centroids: c1 = 2; c2 = (4+10+12+3+20+30+11+25)/8 = 115/8 = 14.375.
• Iteration 2: points closer to 2 → {2, 3, 4}; closer to 14.375 → {10, 11, 12, 20, 25, 30}.
  • c1 = (2+3+4)/3 = 3; c2 = (10+11+12+20+25+30)/6 = 108/6 = 18.
• Iteration 3: |10−3| = 7 vs |10−18| = 8 → 10 joins c1; |11−3| = 8 vs |11−18| = 7 → stays c2.
  • Clusters {2,3,4,10} and {11,12,20,25,30} → c1 = 19/4 = 4.75, c2 = 98/5 = 19.6.
• Iteration 4: |10−4.75| = 5.25 vs |10−19.6| = 9.6 → c1; |11−4.75| = 6.25 vs |11−19.6| = 8.6 → c1 now; |12−4.75| = 7.25 vs 7.6 → c1.
  • Clusters {2,3,4,10,11,12} and {20,25,30} → c1 = 42/6 = 7, c2 = 75/3 = 25.
• Iteration 5: assignments unchanged (12 is 5 from 7, 13 from 25; 20 is 13 from 7, 5 from 25) → converged: {2,3,4,10,11,12} with centroid 7 and {20,25,30} with centroid 25.
• SSE = (25+16+9+9+16+25) + (25+0+25) = 100 + 50 = 150.

Example 4 - Confusion matrix for a fraud model (10,000 UPI transactions, 100 actually fraudulent)

• Accuracy = (70 + 9,670)/10,000 = 97.4% - yet predicting "genuine" always gives 99.0%, so accuracy is the wrong headline.
• Precision = 70/(70+230) = 70/300 = 23.3% (of alerts raised, under a quarter are real).
• Recall = 70/(70+30) = 70% (of real frauds, 70% caught).
• F1 = 2(0.2333×0.70)/(0.2333+0.70) = 0.3267/0.9333 = 35.0%.
• Specificity = 9,670/(9,670+230) = 97.7%; false positive rate = 2.3%.
• Business reading: 230 genuine customers blocked per 10,000 transactions. Raise the score threshold to lift precision, accepting lower recall - the trade-off is read off the ROC/precision-recall curve.

Example 5 - Naive Bayes with Laplace smoothing

Training: 100 SMS, 40 spam and 60 ham. The word "lottery" appears in 12 spam and 1 ham; "offer" in 20 spam and 6 ham.

• Priors: P(spam) = 0.40, P(ham) = 0.60.
• P(lottery|spam) = 12/40 = 0.30; P(offer|spam) = 20/40 = 0.50; P(lottery|ham) = 1/60 = 0.0167; P(offer|ham) = 6/60 = 0.10.
• Message = {lottery, offer}: score(spam) = 0.40 × 0.30 × 0.50 = 0.060; score(ham) = 0.60 × 0.0167 × 0.10 = 0.001.
• P(spam|X) = 0.060/(0.060+0.001) = 98.4% → classify as spam.
• If "lottery" had appeared in 0 ham messages, P(lottery|ham) = 0 would zero the whole product; Laplace smoothing with v = 2 (present/absent) gives (0+1)/(60+2) = 0.0161 instead.

Where it is used

• Banking, NBFC and credit bureaus: application and behavioural scorecards, early-warning and NPA prediction, AML/transaction monitoring - HDFC Bank, ICICI, Bajaj Finserv, CIBIL/Experian; RBI expects explainable, auditable models, so logistic regression and trees still dominate over black boxes.
• E-commerce and consumer internet: market-basket and recommendation (Flipkart, Myntra, BigBasket, Swiggy, Zomato), search ranking, dynamic pricing, return-fraud detection, A/B evaluation.
• Telecom: churn prediction, next-best-offer, network fault and anomaly mining on CDRs - Jio, Airtel, Vi; roles titled Data Analyst / Decision Scientist.
• Payments and government data: fraud and anomaly detection on UPI at NPCI and member banks, GST return analytics for evasion and circular-trading detection at GSTN, PDS and scheme-targeting analytics - usually on anonymised or aggregated data to stay outside DPDP personal-data obligations.
• Healthcare, insurance and pharma: claim-fraud detection, disease-risk clustering, hospital readmission prediction; Ayushman Bharat claim analytics.
• Job roles that test this topic: Data Analyst, Data Scientist, BI/Analytics Engineer, Risk Analyst, Fraud Analyst and Decision Scientist at TCS, Infosys, Wipro, Accenture, Deloitte, Mu Sigma, Fractal, ZS Associates - interviews typically ask one SQL round, one statistics/mining round and one case round.

Interview questions

• Q: How is data mining different from KDD? A: KDD is the complete process from data selection and cleaning to pattern evaluation and presentation; data mining is the single step inside it that applies an algorithm to extract patterns.
• Q: How does data mining differ from OLAP? A: OLAP is verification-driven - the user poses a hypothesis and slices the cube; mining is discovery-driven and produces hypotheses and models the user did not specify.
• Q: State the Apriori property and why it matters. A: Every subset of a frequent itemset is frequent, so if an itemset is infrequent none of its supersets can be frequent - this prunes candidate generation and is the only reason Apriori is tractable.
• Q: A rule has 90% confidence. Is it interesting? A: Not necessarily - if the consequent occurs in 90% of all transactions anyway, lift is 1 and the rule carries no information; always check lift or another correlation measure.
• Q: Why does C4.5 use gain ratio instead of information gain? A: Information gain is biased towards attributes with many distinct values (an ID attribute gives maximal gain); gain ratio divides by the split information to penalise such attributes.
• Q: When do you need to normalise features? A: For distance- or gradient-based methods - k-NN, k-means, SVM, PCA, neural networks. Decision trees, random forests and Naive Bayes are scale-invariant and do not need it.
• Q: Why is k-means sensitive to initial centroids, and what is the fix? A: It is a local-optimum hill-climber on SSE, so poor seeds give poor clusters; run it multiple times with different seeds or use k-means++ seeding.
• Q: Your fraud model is 99% accurate. Why is management unimpressed? A: With 1% fraud, predicting "genuine" always is 99% accurate; the model must be judged on recall, precision, F1 and AUC on the minority class.
• Q: What is overfitting and how do you detect it? A: The model learns training-set noise and generalises poorly; detect it as a large gap between training and validation accuracy, and treat it with pruning, regularisation, fewer features, more data or cross-validation.
• Q: Why is "Naive" Bayes naive, and why does it still work? A: It assumes all attributes are conditionally independent given the class, which is rarely true; the class ranking often survives the wrong probabilities, so classification accuracy remains good even when the estimates are poor.
• Q: Difference between classification and clustering? A: Classification is supervised - it learns from labelled data to predict a known class; clustering is unsupervised - it discovers groups with no labels, and the groups must be interpreted afterwards.
• Q: What privacy constraints apply to mining customer data in India? A: The DPDP Act, 2023 requires lawful consent, purpose limitation and data minimisation for digital personal data, with the Rules phasing the main fiduciary duties in from 13 May 2027 (IT Act section 43A and the SPDI Rules apply until then); anonymised and aggregated data falls outside the Act, and techniques such as k-anonymity and differential privacy are used to get there. Confirm the current notification status before quoting it.

Common mistakes

• Confusing support with confidence, or computing confidence as support(X ∪ Y)/support(Y) instead of /support(X) - the denominator is always the antecedent.
• Forgetting the rule-generation phase of Apriori: finding frequent itemsets is only half the algorithm, and a frequent itemset is not itself a rule.
• Using log base e instead of log base 2 in entropy, so information gain is in nats and the textbook answer never matches.
• Reporting accuracy alone on imbalanced data, and ignoring precision, recall and the cost asymmetry between FP and FN.
• Running k-means on unnormalised features, so an attribute measured in rupees swamps one measured in years.
• Letting test data leak into preprocessing - fitting the scaler, imputer or feature selector on the full data set before splitting - which inflates reported accuracy.
• Treating a discovered correlation as causation and acting on it without an experiment; and deleting outliers by reflex when, in fraud or fault detection, the outliers are the target.

Related topics

Data warehousing and ETL · Machine learning workflow · Database management systems and SQL · Big data analytics with Hadoop and Spark · Statistics and probability for data science · Information security fundamentals