Transfer learning and fine-tuning

In short

Transfer learning reuses the representations a network has already learnt on a large source task (ImageNet, a web-scale text corpus) to solve a different but related target task with far less data and compute. Fine-tuning is the main technique: keep the pretrained weights as the initialisation, replace the output head, and continue training on the target data at a small learning rate. It matters because almost no Indian industry project has millions of labelled examples — a crop-disease app, a chest X-ray classifier or a Hindi-English support-ticket classifier typically has a few thousand labels, and training from scratch then overfits badly. It is examined in AI/DS and CSE deep-learning papers (types of transfer, freezing strategies, catastrophic forgetting, parameter-efficient tuning) and appears in GATE DA-style questions on trainable-parameter counts and in nearly every ML placement interview (TCS, Infosys, Wipro, Fractal, Mu Sigma, Qure.ai, Sarvam AI, Flipkart, Jio).

Explained

Definitions and formal setting

• A domain D = {X, P(X)}: the feature space plus the input distribution. A task T = {Y, f(·)}: the label space plus the predictive function.
• Transfer learning: given a source domain/task (Ds, Ts) and a target (Dt, Tt), with Ds ≠ Dt or Ts ≠ Tt, improve learning of ft using knowledge from the source. Standard supervised learning is the special case Ds = Dt, Ts = Tt.
• Domain shift (covariate shift): P(X) differs but the labels mean the same thing — ImageNet photos vs mobile-camera leaf photos in a field.
• Task shift: label space differs — 1,000 ImageNet classes → 6 crop diseases.
• What gets transferred: instances (re-weighted source samples), features/representations (the common case — convolutional filters, token embeddings), parameters (weight initialisation) or relational knowledge.
• Classification by label availability: inductive (target labels available — fine-tuning), transductive (source labels only; target unlabelled — domain adaptation), unsupervised (neither; clustering/dimensionality reduction).
• Related but distinct: multi-task learning (tasks trained together, shared trunk), self-supervised pretraining (the source task is artificial — masked-token prediction, contrastive views), continual/lifelong learning (a sequence of tasks without forgetting), few/zero-shot learning (very few or no target labels).

Why pretrained features transfer: the layer hierarchy

• Early CNN layers learn generic features — edges, colour blobs, Gabor-like filters — which are near-universal for natural images.
• Middle layers learn textures and parts (wheels, eyes, leaf veins); late layers learn task-specific object compositions tied to the source label set.
• Transferability therefore decreases with depth: the deeper the layer, the more source-specific it is, so the deeper layers are the ones you replace or re-train.
• In Transformer language models, lower layers encode surface and syntactic information, middle layers syntax/semantics, upper layers task-oriented semantics; the [CLS]/pooled vector is the usual sentence representation.
• Two practical consequences: (a) freeze from the bottom, train from the top; (b) a target task very unlike the source (ImageNet → radar/seismic signals) transfers less from the top layers but still benefits from the bottom ones.

The fine-tuning recipe, step by step

1. Choose a backbone matched to the modality and budget: ResNet/EfficientNet/ViT or ConvNeXt for vision; BERT/RoBERTa/DeBERTa, IndicBERT or MuRIL for Indian-language text; Whisper for speech; a Llama/Mistral/Gemma-class decoder for generation. Check the licence (Apache-2.0 vs a community licence with use restrictions).
2. Match the preprocessing exactly to pretraining: input size (224×224), the same mean/std normalisation, the same tokenizer and vocabulary. A mismatch silently destroys accuracy.
3. Replace the head: drop the source classifier and attach a new linear layer of size (feature dim + 1) × C for C target classes (or a regression head, CRF, segmentation decoder).
4. Decide what to train — the four options below.
5. Warm up the head first: train only the new head for 1–3 epochs so its random gradients do not corrupt good backbone weights, then unfreeze.
6. Use a small learning rate for pretrained weights: typically 1e-5 to 5e-5 for BERT-class models (2–4 epochs, batch 16–32) and 1e-4 to 1e-3 for a CNN head, with linear warmup then cosine or step decay, plus weight decay and early stopping on a validation set.
7. Regularise for small data: augmentation (flip, crop, colour jitter, RandAugment, mixup; back-translation or synonym swap for text), dropout, label smoothing, layer-wise decay, and freeze BatchNorm statistics.
8. Evaluate on a held-out target set with the metric that matters (macro-F1 or AUROC under class imbalance, not accuracy), and compare against a from-scratch baseline and a frozen-features baseline.

Four strategies, and how to choose

• Feature extraction (linear probing): freeze the whole backbone, train only the head. Features can be computed once and cached — extremely cheap, no GPU needed for the head, no risk of forgetting. Best when target data is small and similar to the source.
• Partial fine-tuning: freeze the lower k blocks, train the top blocks + head. The usual middle ground.
• Full fine-tuning: update every weight at a low learning rate. Highest accuracy when target data is reasonably large; costliest in memory and most prone to overfitting and forgetting.
• Parameter-efficient fine-tuning (PEFT): keep the backbone frozen and train a small number of injected parameters —
  • LoRA: add a low-rank update ΔW = BA to chosen weight matrices (usually the attention projections); QLoRA does the same over a 4-bit quantised base model, which is how a 7B model is tuned on a single 16 GB GPU or Colab.
  • Adapters: small bottleneck MLPs inserted after each sublayer.
  • Prefix/prompt tuning (P-tuning): learn continuous vectors prepended to the keys/values or the input embeddings.
  • BitFit: train only the bias terms.
  • Advantages: 0.01–1% trainable parameters, one shared base model with many swappable task deltas (a 20 MB LoRA file per client), far less optimiser memory. Limitation: usually a small accuracy gap versus full fine-tuning on large, very different target datasets, and added inference latency unless the LoRA is merged back into W.
• Classic decision matrix:

Problems: forgetting, negative transfer, shift and leakage

• Catastrophic forgetting: fine-tuning overwrites source knowledge, so the model loses general ability (a chat model fine-tuned hard on one narrow task stops following general instructions). Mitigations: smaller LR, fewer epochs, freeze more layers, rehearsal (mix a slice of source/general data back in), EWC (penalise movement of parameters with high Fisher information), PEFT (base weights untouched).
• Negative transfer: the source actually hurts — unrelated domains, or a tiny target set with a huge backbone. Detect it by always running a from-scratch and a frozen-feature baseline.
• BatchNorm trap: in a frozen backbone, BN layers in training mode keep updating running mean/variance from target batches, which changes the "frozen" function. Set the module to eval mode (or use GroupNorm/LayerNorm backbones). Conversely, re-estimating BN statistics on target data alone (AdaBN) is a cheap domain-adaptation trick.
• Unsupervised domain adaptation when target labels are absent: align feature distributions with MMD, CORAL (second-order statistics), or adversarially with a domain-adversarial network (DANN) using a gradient-reversal layer; or pseudo-labelling/self-training with confidence thresholds.
• Knowledge distillation as transfer: a large fine-tuned teacher trains a small student on softened probabilities — used to ship a 6-layer model to a mobile app (DistilBERT, MobileBERT).
• Data leakage and evaluation: augment/split *before* feature caching; group-split by patient, farm or user, not by image, or the reported accuracy is inflated.
• Licence, privacy and compliance (India): pretrained weights carry licences that may restrict commercial use; fine-tuning on personal data (health records, KYC images, call recordings) attracts the Digital Personal Data Protection Act, 2023 — consent, purpose limitation, data-minimisation and breach reporting (its rules were notified in 2025 with phased timelines, so check the current stage before relying on dates). Medical AI additionally follows CDSCO software-as-a-medical-device requirements and ICMR's ethical guidelines for AI in biomedical research. Models can memorise training text, so de-identify before fine-tuning.

Fine-tuning large language models

• Stages: self-supervised pretraining → continued/domain-adaptive pretraining (more unlabelled in-domain text, e.g. legal or Indic corpora) → supervised fine-tuning (SFT) on instruction–response pairs → preference alignment (RLHF or DPO).
• Instruction fine-tuning needs only thousands of high-quality examples; quality and diversity beat volume.
• Alternatives before you fine-tune: prompt engineering, few-shot in-context examples, and RAG (retrieval-augmented generation). Rule of thumb: RAG for *knowledge* that changes, fine-tuning for *behaviour, format, style and domain jargon*.
• Memory arithmetic decides feasibility: full Adam fine-tuning needs roughly 16–20 bytes per parameter (weights + gradients + two Adam moments + master copy); LoRA/QLoRA needs that only for the adapter, plus the frozen quantised base.
• Tooling commonly asked about: Hugging Face transformers + peft + bitsandbytes, timm for vision backbones, PyTorch Lightning, and mixed precision + gradient checkpointing + gradient accumulation to fit a larger effective batch.

Key points

• Transfer learning – reuse source-task knowledge to learn a related target task with less data and compute.
• Fine-tuning – initialise from pretrained weights, replace the head, continue training at a low learning rate.
• Feature extraction – backbone frozen, only the head trains; features can be cached once.
• Transferability falls with depth – early layers generic, late layers source-specific; freeze bottom, train top.
• Head replacement – new linear layer with (d + 1) × C parameters for C classes.
• Head warm-up then unfreeze – prevents random head gradients from damaging good features.
• Discriminative / layer-wise learning rates – smaller LR for lower layers, larger for the head.
• Typical BERT recipe – LR 2e-5 to 5e-5, 2–4 epochs, batch 16–32, linear warmup and decay.
• Catastrophic forgetting – loss of source ability; countered by low LR, rehearsal, EWC or PEFT.
• Negative transfer – unrelated source hurts; always keep a from-scratch baseline.
• LoRA – ΔW = BA with rank r ≪ d; trainable r(d + k) per matrix, scaled by α/r; mergeable at inference.
• QLoRA – LoRA over a 4-bit quantised base, enabling 7B-class tuning on one consumer GPU.
• Adapters / prefix tuning / BitFit – other PEFT families; 0.01–1% of parameters trained.
• Domain adaptation – align unlabelled target features via MMD, CORAL, DANN or pseudo-labelling.
• AdaBN / BN eval mode – re-estimate or freeze BatchNorm statistics; a classic silent bug.
• Knowledge distillation – soft teacher probabilities at temperature T train a smaller student.
• RAG vs fine-tuning – RAG supplies changing facts; fine-tuning teaches format, style and domain behaviour.
• Compliance – model licences plus the DPDP Act, 2023 govern fine-tuning on Indian personal data.

Formulas and examples

Example 1 — Trainable parameters and compute saving (ResNet-50, 5 crop diseases).

• ResNet-50 has ≈ 25.56 M parameters, of which the 1000-class head is 2048 × 1000 + 1000 = 2,049,000.
• Remove it and attach a 5-class head: P_head = 2048 × 5 + 5 = 10,245. New total ≈ 25.56 M − 2.049 M + 0.010 M ≈ 23.52 M.
• Feature extraction: φ = 10,245 / 23.52 M = 0.044 % trainable.
• Fine-tune layer4 + head instead: layer4 ≈ 15.0 M parameters → φ ≈ (15.0 + 0.01)/23.52 ≈ 64 %.
• Compute: forward pass ≈ 4.1 GFLOPs per 224×224 image, backward ≈ 2× forward → 12.3 GFLOPs for full training. For 2,000 images × 20 epochs: full fine-tune ≈ 2,000 × 20 × 12.3 = 492 TFLOPs; cached features ≈ 2,000 × 4.1 = 8.2 TFLOPs once, i.e. 60× less (492/8.2 = 60), with the head costing almost nothing.

Example 2 — LoRA parameter count and memory (7B decoder).

• Model dimension d = k = 4096, 32 layers, LoRA on q, k, v, o projections, rank r = 8.
• Per matrix: r(d + k) = 8 × 8192 = 65,536. Per layer: 4 × 65,536 = 262,144. All layers: 32 × 262,144 = 8,388,608 ≈ 8.39 M → φ = 8.39 M / 7 B ≈ 0.12 %.
• Adam memory at 16 bytes per trainable parameter: full fine-tune ≈ 7 × 10⁹ × 16 = 112 GB (needs multi-GPU); LoRA ≈ 8.39 × 10⁶ × 16 ≈ 134 MB, plus the 4-bit frozen base ≈ 7 × 10⁹ × 0.5 B ≈ 3.5 GB → fits a single 16 GB GPU with activations.
• The saved adapter file is a few tens of MB, so one base model can serve many clients' tasks.

Example 3 — Discriminative learning rates. Head LR 1 × 10⁻³, divisor η = 2.6 per group downward:

(1e-3/2.6 = 3.85e-4; /2.6 = 1.48e-4; /2.6 = 5.69e-5.) Early layers barely move, protecting generic filters.

Example 4 — Distillation temperature. Teacher logits z = (4, 2, 1).

• T = 1: e⁴ = 54.598, e² = 7.389, e¹ = 2.718; sum = 64.705 → probabilities 0.844, 0.114, 0.042.
• T = 2: e² = 7.389, e¹ = 2.718, e⁰·⁵ = 1.649; sum = 11.756 → 0.629, 0.231, 0.140.
• The softened vector carries the teacher's "dark knowledge" (at T = 2, class 2 is about 1.6× as plausible as class 3, against 2.7× at T = 1); the T² factor in the loss keeps gradient magnitudes comparable across temperatures.

Example 5 — Strategy comparison on a small Indian dataset (1,200 leaf images, 6 classes, 200 each; 70/15/15 split; single T4 GPU). Indicative figures from this kind of project, not universal constants:

Reading: with 200 images per class, pretraining is worth ≈ 30 F1 points, and most of the benefit arrives with the cheapest option.

Where it is used

• Agritech and rural apps: crop-disease and pest identification on phone cameras (ImageNet or PlantVillage backbones fine-tuned on local images) in Kisan-advisory and agri-input products.
• Healthcare AI: Qure.ai (chest X-ray, head CT), Niramai, Tricog and hospital AI teams fine-tune vision backbones on Indian patient data under CDSCO/ICMR requirements; labels are scarce and costly, so transfer is mandatory.
• Indian-language NLP: AI4Bharat (IIT Madras) IndicBERT, IndicTrans2 and Google's MuRIL are fine-tuned for sentiment, NER, intent and translation; Bhashini (MeitY) and Sarvam AI build speech and text services for 22 scheduled languages by adapting multilingual pretrained models with Whisper-style fine-tuning for ASR.
• BFSI and services: fine-tuned BERT/LLMs for KYC document extraction, insurance claim triage, complaint routing and RBI-compliance summarisation at banks, NBFCs and IT services firms (TCS, Infosys, Wipro, HCL) — usually LoRA on a frozen base for data-isolation across clients.
• E-commerce and media: Flipkart, Myntra, Meesho and Nykaa fine-tune vision models for catalogue tagging, visual search and counterfeit detection; OTT platforms adapt recommenders and thumbnail/scene models.
• Geospatial and industry: ISRO/NRSC Bhuvan land-use and change-detection models adapted from RGB pretraining to multispectral bands; defect detection on manufacturing lines with a few hundred defect images.
• Job roles: ML engineer, applied scientist, computer-vision engineer, NLP engineer, MLOps — fine-tuning, PEFT and evaluation are standard screening topics.

Interview questions

• Q: What is the difference between transfer learning and fine-tuning? A: Transfer learning is the general idea of reusing source-task knowledge; fine-tuning is one implementation of it in which pretrained weights are used as the initialisation and then updated on the target data.
• Q: Why do we freeze early layers and train later ones, not the reverse? A: Early layers learn generic edge/texture features that transfer to almost any natural image, while later layers are specific to the source labels and must be re-learnt for the new task.
• Q: When would you prefer feature extraction over full fine-tuning? A: When target data is small and similar to the source, or compute is limited — it avoids overfitting and forgetting, and features can be cached once for very fast training.
• Q: Why use a much smaller learning rate when fine-tuning? A: The weights are already near a good solution; a large LR would take large steps that destroy pretrained features, which is catastrophic forgetting.
• Q: What is catastrophic forgetting and how do you control it? A: The model loses previously learnt ability while adapting to the new task; reduce the LR, train fewer epochs, freeze more layers, rehearse old data, add an EWC penalty, or use PEFT so base weights never change.
• Q: Explain LoRA and why it saves so much memory. A: It freezes W and learns a low-rank update BA with rank r ≪ d, so only r(d + k) parameters per matrix are trained — optimiser states and gradients are needed for ~0.1 % of the weights instead of all of them.
• Q: Is there an inference penalty with LoRA? A: Only if the adapter is kept separate; B·A can be merged into W after training, giving identical latency to the original model.
• Q: What is negative transfer and how do you detect it? A: When the source knowledge hurts target performance because the domains are unrelated; detect it by comparing against from-scratch and frozen-feature baselines on the same validation split.
• Q: A frozen ResNet still changes its predictions between epochs. Why? A: BatchNorm layers left in training mode keep updating their running mean and variance from target batches; put them in eval mode or freeze the statistics.
• Q: Fine-tuning or RAG for a company FAQ bot? A: RAG, because the knowledge changes and must be cited; fine-tune only to fix tone, output format or domain vocabulary, often both together.
• Q: You have 300 labelled images for 10 classes. What is your plan? A: Pretrained backbone with frozen features plus a linear or SVM head, strong augmentation, stratified cross-validation, macro-F1 as the metric, and only then try unfreezing the top block.
• Q: Which hyperparameters matter most in fine-tuning a BERT-class model? A: Learning rate (2e-5 to 5e-5), number of epochs (2–4, since it overfits quickly), batch size, warmup fraction and maximum sequence length; seed variance is also large on small datasets.

Common mistakes

• Changing preprocessing — different input size, normalisation statistics or tokenizer — from what the model was pretrained with, then blaming the model.
• Using the same large learning rate for backbone and new head, or skipping head warm-up, so good features are wrecked in the first few steps.
• Forgetting model.eval() / frozen BatchNorm statistics, or forgetting to turn dropout off at inference, and reporting unstable metrics.
• Leaving requires_grad = True everywhere while claiming the backbone is frozen, or passing frozen parameters to the optimiser and wondering why memory is unchanged.
• Augmenting or caching features before the train/validation split, or splitting by image instead of by patient/farm/user — leakage that inflates accuracy.
• Reporting plain accuracy on an imbalanced target set instead of macro-F1, AUROC or a confusion matrix, and omitting the from-scratch baseline that proves transfer helped.
• Over-training on a small dataset (20 epochs where 3 suffice) and treating the resulting overfit curve as a model-capacity problem.

Related topics

Convolutional neural networks · Transformers and attention · Training deep networks: optimisers, normalisation and regularisation · Neural networks and backpropagation · Model compression and knowledge distillation · MLOps and model deployment