AI ethics, bias and responsible AI

In short

AI ethics is the study of what an automated decision system ought and ought not do; bias is the measurable part of it — a model that is systematically less accurate or less favourable for one group of people than another. Responsible AI is the engineering practice that makes ethics operational: documented data, measured fairness, explainable predictions, privacy protection, human oversight and audit trails built into the pipeline rather than bolted on. It matters because AI/DS graduates in India now build credit-scoring, hiring-shortlist, insurance-pricing, fraud and welfare-eligibility models where a biased output has legal and human consequences under the DPDP Act, 2023, RBI and SEBI supervisory expectations and NITI Aayog's Responsible AI principles. It is examined in AICTE AI&DS electives ("Ethics in AI", "Responsible AI"), in UGC CBCS data-science papers, and asked in placement rounds at TCS, Infosys, Wipro, Accenture, Fractal, Mu Sigma, ZS, Swiggy, Razorpay, HDFC and every bank's analytics unit. Expect numerical questions on fairness metrics computed from confusion matrices — that is the most examinable part of the topic.

Explained

What "bias" means, and the five places it enters

• Statistical bias (error of estimate) is not the same as social/discriminatory bias (unequal treatment of groups). Exams ask both; keep the words apart.
• Historical bias: the world the data came from was already unequal, so a perfectly accurate model reproduces it. A hiring model trained on 10 years of resumes from a male-dominated team learns "male" as a success signal.
• Representation (sampling) bias: some groups are under-sampled. A face dataset dominated by lighter-skinned faces gives high overall accuracy and poor accuracy for darker-skinned faces.
• Measurement (label) bias: the proxy label is not the thing you care about. "Arrested" is a proxy for "committed crime"; "clicked" is a proxy for "liked"; "spend" is a proxy for "creditworthy".
• Aggregation bias: one model for all subgroups when their relationships differ — e.g. HbA1c thresholds for diabetes behave differently across ethnic groups, so a single cut-off misdiagnoses one of them.
• Deployment / feedback bias: the model changes the world it then learns from. Predictive policing sends patrols to area A, more arrests are recorded in A, the next model is more certain about A. Also called a runaway feedback loop.
• Proxy variables: removing the protected attribute does not remove the bias. PIN code proxies for caste/religion/community in Indian cities, college name proxies for socio-economic class, mobile handset price proxies for income.

Fairness definitions (and why you cannot have them all)

Let A = protected attribute (group), Y = true label, Ŷ = prediction, S = score.

• Fairness through unawareness: do not use A as a feature. Weakest notion — defeated by proxies.
• Group fairness / demographic parity (statistical parity): P(Ŷ=1 | A=a) equal across groups. Equal selection rates, ignores whether Y differs across groups.
• Disparate impact ratio (adverse impact ratio): ratio of selection rates. The US "four-fifths rule" treats < 0.80 as evidence of adverse impact; it is a widely used audit heuristic, not Indian law.
• Equal opportunity: equal TPR (recall) across groups — equally good at catching the qualified/positive cases.
• Equalised odds: equal TPR and FPR across groups.
• Predictive parity (outcome test): equal precision (PPV) across groups — of those we flagged, the same share are genuinely positive.
• Calibration within groups: among those scored p, the actual positive rate is p, in every group.
• Individual fairness: similar individuals get similar outcomes (needs a similarity metric — hard in practice).
• Counterfactual fairness: the prediction is unchanged if the individual's protected attribute (and its causal descendants) were changed.
• The impossibility result (Kleinberg–Mullainathan–Raghavan; Chouldechova): if base rates differ across groups and the classifier is not perfect, you cannot simultaneously have calibration, equal FPR and equal FNR. This is the core of the COMPAS dispute — ProPublica measured FPR gaps, Northpointe measured predictive parity; both were arithmetically right.
• Consequence for exams and interviews: fairness is a *choice*, justified by context. Punitive decisions (denial, arrest, rejection) → control FPR. Beneficial/opportunity decisions (loan offer, scholarship, screening for a disease) → control FNR/TPR.

Responsible AI principles and the Indian framework

• NITI Aayog "Responsible AI for All" (Approach Document, Part 1 2021, Part 2 on operationalising) sets seven system/societal principles: safety and reliability; equality (non-discrimination); inclusivity and non-discrimination; privacy and security; transparency; accountability; protection and reinforcement of positive human values. IndiaAI Mission (2024 onward) funds Safe & Trusted AI as one of its pillars.
• Digital Personal Data Protection Act, 2023 (DPDP Act): consent must be free, specific, informed and unambiguous with a clear notice; purpose limitation, data minimisation, accuracy, storage limitation; rights to access, correction, erasure and grievance redressal; Significant Data Fiduciaries face extra duties including Data Protection Impact Assessment and independent audit; Consent Managers; penalties up to ₹250 crore per breach category. The DPDP Rules were notified in 2025 with phased commencement — check the current notified phase before quoting a compliance date, as the rollout runs in stages.
• Sector regulators: RBI's Digital Lending guidelines (no automated credit limit increase without consent, Key Fact Statement, data stored in India), RBI's FREE-AI committee work on AI in finance, SEBI's rules requiring intermediaries to report and remain responsible for AI/ML tools they use, IRDAI's guidance on data use in insurance, and the MeitY-notified IT Rules on intermediary due diligence and deepfake/synthetic-media labelling. The 2026 position on mandatory AI-content labelling has been evolving — state the requirement in principle, not a clause number, unless you have checked the current notification.
• No standalone "AI Act" in India as of late 2026; governance works through DPDP, sectoral regulators, the Information Technology Act and advisories. The EU AI Act (risk tiers: unacceptable / high / limited / minimal, in force from 2024 with staged obligations) is the comparator usually asked for.
• Standards: ISO/IEC 42001:2023 (AI management system — certifiable), ISO/IEC 23894:2023 (AI risk management), ISO/IEC TR 24027 (bias in AI), NIST AI Risk Management Framework 1.0 (Govern, Map, Measure, Manage), IEEE 7000-series on ethical design. BIS has an AI standardisation committee (LITD 30) issuing Indian Standards aligned to ISO/IEC work.
• Professional duty: an engineer's signature on a model carries the same obligation of competence and public interest as on a structural design.

How bias is found: the measurement workflow

1. Define the decision and the harm — allocative (who gets the loan) or representational (who is depicted how). Name the groups and get legal sign-off on using the protected attribute for *testing*.
2. Document the data — Datasheets for Datasets: provenance, collection method, consent basis, population covered, known gaps, time window.
3. Slice the metrics — never report one aggregate number. Build a per-group confusion matrix and compute accuracy, TPR, FPR, PPV, selection rate.
4. Disaggregated and intersectional evaluation — test A alone, B alone and A∩B. Gender Shades (Buolamwini and Gebru, 2018) found commercial gender classifiers erred ~0.8% on lighter males and up to ~34.7% on darker females; neither single axis exposed that.
5. Test with a gap threshold agreed in advance (e.g. TPR gap ≤ 5 pp, disparate impact ≥ 0.8) so the result is not negotiated after the fact.
6. Monitor post-deployment — drift in data and in the fairness gap, complaint/appeal rates per group, override rates by human reviewers.
7. Tools: IBM AI Fairness 360, Microsoft Fairlearn, Google What-If Tool, Aequitas, fairlearn.metrics.MetricFrame.

How bias is reduced: three families of mitigation

• Pre-processing (fix the data): reweighing (give instance weights so group and label become independent), resampling/oversampling under-represented groups, disparate impact remover (repair feature distributions), learning fair representations, targeted data collection — usually the most durable fix.
• In-processing (fix the training): fairness constraints in the objective (exponentiated gradient / reductions approach), adversarial debiasing (an adversary tries to predict A from the representation; the encoder is trained to defeat it), prejudice remover regularisation, group-DRO for worst-group accuracy.
• Post-processing (fix the outputs): group-specific thresholds (Hardt et al. equalised-odds post-processing), reject-option classification, calibrated-equalised-odds. Cheapest and model-agnostic, but it *uses the protected attribute at decision time* — in India that can itself be legally and ethically contentious, so it needs counsel.
• Fairness–accuracy trade-off: mitigation usually costs a little overall accuracy. Quantify it (e.g. "AUC 0.84 → 0.83, TPR gap 18 pp → 4 pp") and let the business and legal owners decide; do not choose silently.

Transparency, explainability and accountability

• Interpretable by design vs post-hoc explanation: logistic regression, scorecards, decision trees, GAMs and rule lists are inherently readable; use them where the decision is high-stakes and the accuracy gap is small.
• Post-hoc, global: permutation feature importance, partial dependence plots, surrogate trees. Post-hoc, local: LIME (local linear surrogate on perturbed samples), SHAP (Shapley values — the only attribution satisfying efficiency, symmetry, dummy and additivity), counterfactual explanations ("your loan would be approved with income ₹8,000 higher"), anchors. Counterfactuals are usually the most *actionable* for a rejected applicant.
• Limitations: explanations are approximations, can be unstable, can be gamed ("fairwashing"), and SHAP on correlated features can mislead. An explanation is not a justification.
• Documentation artefacts: Model Cards (intended use, out-of-scope use, training data, disaggregated performance, ethical considerations), Datasheets for Datasets, FactSheets, system cards, DPIA reports.
• Accountability: a named model owner, a risk tier, a review board, versioned model/data/code lineage, an appeal and grievance channel (DPDP requires one from 13 May 2027), and a kill switch. Human-in-the-loop (human decides), human-on-the-loop (human supervises), human-in-command (human sets the remit) — distinguish them; a human who rubber-stamps 1,000 cases an hour is not meaningful oversight (automation bias).
• Privacy techniques: anonymisation vs pseudonymisation, k-anonymity/l-diversity, differential privacy (ε-budget; lower ε = stronger privacy, more noise), federated learning, secure aggregation, synthetic data. Note the intersection: DP noise degrades accuracy most for small groups, which can *worsen* fairness.
• Generative-AI-specific harms: hallucination, toxicity, stereotype amplification, copyright and training-data consent, deepfakes and non-consensual imagery, prompt injection and jailbreaks, model and data leakage, environmental cost. Controls: RLHF/constitutional alignment, safety classifiers and guardrails, red-teaming, provenance/watermarking (C2PA content credentials), retrieval grounding with citations, evaluation suites for bias and toxicity.

Key points

• Algorithmic bias - systematic difference in a model's error or favourability across groups, not mere statistical noise.
• Fairness through unawareness - dropping the protected attribute; fails because of proxies like PIN code and college name.
• Demographic parity - equal selection rate P(Ŷ=1|A) across groups; ignores true qualification differences.
• Equal opportunity - equal TPR across groups; the right target when a false negative denies a benefit.
• Equalised odds - equal TPR *and* FPR; stricter, usually achieved by post-processing thresholds.
• Predictive parity - equal precision across groups; what a vendor typically claims.
• Impossibility theorem - with unequal base rates and an imperfect model, calibration, equal FPR and equal FNR cannot all hold.
• Disparate impact ratio - selection-rate ratio; < 0.8 is the common audit red flag (US four-fifths rule, used as a heuristic in India).
• Feedback loop - the model's own decisions generate its next training data, amplifying the original skew.
• Intersectional testing - evaluate A∩B subgroups; single-axis audits missed the Gender Shades gap.
• Mitigation families - pre-processing (data), in-processing (objective), post-processing (thresholds).
• Reweighing - instance weights making group and label statistically independent; cheap, model-agnostic pre-processing.
• SHAP - Shapley-value attribution of a prediction to features, additive and locally exact; LIME is a local linear surrogate.
• Model Card / Datasheet - standard documentation of intended use, data provenance and per-group performance.
• DPDP Act, 2023 - India's data protection law: consent, purpose limitation, data-principal rights, DPIA for Significant Data Fiduciaries, penalties to ₹250 crore; Rules notified 2025 in phases.
• NITI Aayog Responsible AI - seven principles including safety, equality, inclusivity, privacy, transparency, accountability, human values.
• ISO/IEC 42001 and NIST AI RMF - certifiable AI management system and the Govern–Map–Measure–Manage risk framework.
• Differential privacy - formal privacy guarantee with budget ε; protects individuals but adds noise that hurts minority-group accuracy.

Formulas and examples

Example 1 — selection rate, SPD and disparate impact in campus shortlisting. A screening model is run on 1,000 applicants from Tier-1 colleges and 1,000 from Tier-2/3 colleges. It shortlists 400 Tier-1 and 240 Tier-2/3.

• SR(Tier-1) = 400/1000 = 0.40; SR(Tier-2/3) = 240/1000 = 0.24.
• SPD = 0.24 − 0.40 = −0.16 (−16 pp).
• DI = 0.24/0.40 = 0.60 — below 0.8, so an adverse-impact flag.
• To reach DI = 0.80 at the same Tier-1 rate, Tier-2/3 shortlists must be 0.8 × 0.40 × 1000 = 320, i.e. 80 more candidates. Only legitimate if their qualification rate supports it — otherwise fix the features (college name is a proxy for class), not just the count.

Example 2 — equal opportunity and equalised odds from confusion matrices. A loan default-risk model (positive = "approve"), two groups, 1,000 applicants each.

• TPR(P) = 420/600 = 0.700; TPR(U) = 180/400 = 0.450 → EOD = 0.450 − 0.700 = −0.25 (−25 pp). Severe equal-opportunity violation.
• FPR(P) = 80/400 = 0.200; FPR(U) = 60/600 = 0.100 → FPR difference = −0.10.
• AOD = ½[(0.100 − 0.200) + (0.450 − 0.700)] = ½(−0.10 − 0.25) = −0.175.
• Precision(P) = 420/500 = 0.840; Precision(U) = 180/240 = 0.750 → predictive parity also fails, but by only 9 pp.
• Accuracy(P) = (420+320)/1000 = 0.740; Accuracy(U) = (180+540)/1000 = 0.720 — nearly equal. Aggregate accuracy hid a 25 pp recall gap: exactly why metrics must be sliced.
• Note the base rates differ (P: 600/1000 = 0.60 positive; U: 400/1000 = 0.40), so by the impossibility result no threshold makes calibration, FPR and FNR all equal.

Example 3 — post-processing with group thresholds. On group U, lowering the score threshold moves 60 applicants from the predicted-negative side into approval; of them 48 are true positives and 12 are false positives.

• New TP(U) = 180 + 48 = 228, FN(U) = 220 − 48 = 172 → TPR(U) = 228/400 = 0.570.
• New FP(U) = 60 + 12 = 72, TN(U) = 540 − 12 = 528 → FPR(U) = 72/600 = 0.120.
• EOD improves from −0.25 to 0.570 − 0.700 = −0.13; accuracy(U) = (228+528)/1000 = 0.756, up from 0.720 here because most of the moved cases were genuine positives.
• Cost check: 12 extra bad loans at ₹2,00,000 average exposure and 60% loss-given-default = 12 × 2,00,000 × 0.60 = ₹14.4 lakh expected loss, against 48 additional good loans. State both numbers; the decision is the business owner's, not the modeller's.

Example 4 — intersectional accuracy, aggregate vs worst group. A face-attribute model is tested on 10,000 images with per-subgroup error rates:

• Overall error = 550/10,000 = 5.5% → accuracy 94.5%, which a datasheet would call "good".
• Worst-group error = 30%, 30× the best group's 1%.
• By single axis: female error = (90+300)/4000 = 9.75%; darker error = (120+300)/3000 = 14.0% — both far below the 30% intersectional figure. The worst harm is invisible to single-axis audits.

Example 5 — differential privacy budget. A query answered with the Laplace mechanism on a count (sensitivity Δf = 1) at ε = 0.5 uses noise scale b = Δf/ε = 1/0.5 = 2, giving standard deviation √2·b ≈ 2.83 counts.

• Over 5 such queries the budget composes to ε_total = 5 × 0.5 = 2.5 (basic composition) — weaker privacy than any single query.
• A subgroup of 50 people absorbs ±2.83 noise on a count of, say, 10 — a ~28% relative distortion — while a 50,000-person group barely notices it. That is how a privacy control can degrade minority-group statistics and worsen fairness.

from fairlearn.metrics import MetricFrame, selection_rate, true_positive_rate
mf = MetricFrame(metrics={"sel": selection_rate, "tpr": true_positive_rate},
                 y_true=y_test, y_pred=y_pred, sensitive_features=group)
print(mf.by_group)          # per-group table
print(mf.difference())      # largest gap
print(mf.ratio())           # disparate impact style ratio

Where it is used

• Banking, NBFCs and digital lending: credit scorecards and collection models must be explainable to RBI, follow the Digital Lending guidelines and Key Fact Statement rules, and withstand reason-code audits — roles like Model Risk Analyst and Model Validation at HDFC, ICICI, Axis, SBI, Bajaj Finserv, Paytm and CRED.
• Insurance and health analytics: IRDAI-supervised pricing and claims models, where PIN code and occupation act as proxies; diagnostic AI (qXR by Qure.ai, Niramai, Tricog) needs per-demographic sensitivity reporting and CDSCO/regulatory evidence.
• Hiring and ed-tech: resume screening, proctoring and video-interview scoring; employers now routinely demand adverse-impact testing before deploying a shortlisting model — relevant to HirePro, Naukri-style platforms and large IT campus hiring.
• Government and welfare: Aadhaar-linked eligibility, DBT targeting, Digital India platforms and urban surveillance, where exclusion errors deny entitlements — audited against DPDP and NITI Aayog principles.
• Platform and ad tech: ranking, recommendation and moderation at Swiggy, Zomato, Flipkart, Meesho, ShareChat and Josh; content moderation for Indic languages where models are weakest, plus IT Rules duties on synthetic media.
• Job roles that own this work: Responsible AI / AI Governance Analyst, Model Risk Manager, ML Engineer (fairness testing in the CI pipeline), Data Protection Officer, internal AI audit at Deloitte, EY, PwC, KPMG, Accenture, TCS and Infosys.

Interview questions

• Q: Will removing gender and caste from the feature list make my model fair? A: No — correlated proxies such as PIN code, college, name and handset price carry the same information; this is "fairness through unawareness" and it is the weakest notion. You still have to measure outcomes per group.
• Q: Why can a model be fair by one definition and unfair by another? A: The impossibility result: when base rates differ across groups and the classifier is imperfect, calibration, equal false-positive rates and equal false-negative rates cannot all hold simultaneously, so you must pick the metric that matches the harm.
• Q: Which fairness metric would you choose for a fraud-detection model? A: Equalised odds, weighted towards equal FPR, because a false positive wrongly blocks a genuine customer; a punitive decision should not misfire more often on one group.
• Q: And for a disease-screening model? A: Equal opportunity (equal TPR/recall), because a false negative denies care; missing cases in one group is the dominant harm.
• Q: Your model has 94% overall accuracy. Is it fine to ship? A: Unknown until it is sliced — in the worked example a 94.5% aggregate hid a 30% error rate on one intersectional subgroup. Report disaggregated and worst-group performance.
• Q: Difference between pre-, in- and post-processing mitigation? A: Pre-processing changes the data (reweighing, resampling), in-processing adds a fairness constraint to training (adversarial debiasing, reductions), post-processing adjusts decisions/thresholds after training. Post-processing is cheapest but uses the protected attribute at decision time.
• Q: SHAP vs LIME? A: SHAP computes Shapley-value attributions that are additive and sum to the prediction minus the baseline, so it is consistent but costlier; LIME fits a local linear surrogate on perturbed samples — fast but less stable and not additively exact.
• Q: Is an explanation enough for accountability? A: No. Accountability needs a named owner, documented intended use and limits, lineage, monitoring, an appeal channel and the ability to turn the system off; an explanation can even be gamed ("fairwashing").
• Q: What does the DPDP Act, 2023 require of an ML team? A: A lawful basis (usually clear, specific consent with notice), purpose limitation and minimisation, honouring access/correction/erasure requests, a grievance channel, and for Significant Data Fiduciaries a DPIA and independent audit; penalties run up to ₹250 crore. These duties apply from 13 May 2027 under the DPDP Rules 2025; until then IT Act s.43A and the SPDI Rules 2011 govern.
• Q: Does India have an AI Act? A: Not as a standalone statute as of late 2026; governance comes from the DPDP Act, the IT Act and IT Rules, sectoral regulators (RBI, SEBI, IRDAI) and NITI Aayog's Responsible AI principles, with the EU AI Act's risk-tier model as the usual comparator.
• Q: How can a privacy control make fairness worse? A: Differential-privacy noise and aggressive anonymisation distort small groups proportionally far more than large ones, degrading accuracy and measurability exactly where representation is already thin.
• Q: What is a feedback loop, with an example? A: The model's decisions shape the data it is retrained on — patrol more in one area, record more arrests there, and the next model is more confident about that area, independent of any change in actual crime.
• Q: What goes in a Model Card? A: Intended and out-of-scope use, training and evaluation data, disaggregated performance by group, fairness metrics and thresholds, known limitations, ethical considerations and the owner/version.

Common mistakes

• Reporting a single aggregate accuracy or AUC and calling the model validated — the gap always lives in the slices.
• Confusing statistical bias (estimation error) with discriminatory bias (group harm), or bias with variance, in a definition question.
• Claiming a model is fair because the protected attribute was dropped, ignoring proxies.
• Computing TPR with the wrong denominator — TPR divides by (TP+FN) within the group, not by the group total or by TP+FP.
• Asserting that all fairness metrics can be satisfied at once; the impossibility result is a standard follow-up question.
• Mitigating bias with group-specific thresholds without checking whether using the protected attribute at decision time is lawful and acceptable.
• Treating ethics as a final sign-off step instead of design-time choices (problem framing, label definition, data collection, monitoring).
• Quoting the "80% rule" or the EU AI Act as Indian law, or giving a DPDP Rules compliance date without checking the current phased notification.

Related topics

MLOps and model deployment · AI regulation and governance · Data pipelines: Airflow and Kafka · Data warehousing and ETL · Explainable AI and model interpretability · Machine learning evaluation metrics