Multimodal AI

In short

Multimodal AI builds models that take in and relate more than one kind of data — text, images, video, speech, tabular signals, sensor streams — in a single shared representation, instead of one model per data type. It matters because real tasks are multimodal: a Flipkart product has a photo, a title and reviews; a doctor's note sits beside an X-ray; a bank's KYC check compares a selfie with an Aadhaar card photo. It is examined in AI/DS electives on advanced deep learning (fusion strategies, contrastive objectives, attention-based alignment) and in GATE DA-style questions on embeddings, similarity and loss functions. Indian employers — Google India, Microsoft IDC, Amazon, Flipkart, Jio, Sarvam AI, Krutrim, AI4Bharat, Niramai, Qure.ai — ask it in placement rounds, usually around CLIP, VQA, fusion levels and handling missing modalities.

Explained

Modalities, and why combining them helps

• A modality is a channel of sensing or representation: vision (image, video), language (text), audio (speech, music), and others — tabular, time series, LiDAR/depth, EEG, touch.
• Heterogeneity gap: modalities differ in dimensionality, sampling rate, structure and noise. An image is a dense 2-D grid of pixels; text is a short discrete symbol sequence. They cannot be concatenated naively and must be projected into comparable spaces.
• Two benefits, examined separately: complementarity (one modality carries information the other lacks — lip movement disambiguates noisy speech) and redundancy (overlapping information gives robustness when one channel degrades).
• Core challenges (standard five, from Baltrušaitis's taxonomy, widely used in syllabi):
  1. Representation — encode several modalities into a joint or coordinated space.
  2. Alignment — find correspondences (which word refers to which image region; which audio frame matches which video frame).
  3. Fusion — combine information to make one prediction.
  4. Translation — map one modality to another (image → caption, text → image, speech → text).
  5. Co-learning — transfer knowledge from a data-rich modality to a data-poor one.

Representation: joint versus coordinated

• Joint (shared) representation: all modalities are mapped into *one* vector space, usually by concatenating or summing encoder outputs and passing through shared layers. Needs all modalities at inference. Used in classification and VQA.
• Coordinated representation: each modality keeps its own encoder and space, but the spaces are *constrained* to agree under a similarity measure (cosine similarity, canonical correlation). Used for retrieval, because text and image can be embedded independently and compared later.
• Encoders in practice: text → Transformer/BERT-style token embeddings; image → Vision Transformer (ViT) patch embeddings or a CNN (ResNet); audio → log-mel spectrogram into a Transformer or CNN; video → frame features plus temporal attention.
• Tokenisation trick that unified the field: split an image into fixed patches (say 16×16), linearly project each patch to a vector, and treat it as a token. Audio spectrogram frames likewise. Then one Transformer can ingest any modality as a token sequence.
• Canonical Correlation Analysis (CCA) is the classical, pre-deep coordinated method: find linear projections of two views that maximise correlation. Deep CCA replaces the linear maps with networks. Often asked as "the statistical ancestor of CLIP".

Fusion strategies (the most examined classification)

• Early fusion (feature-level): concatenate raw or low-level features before the model. Captures low-level interaction; suffers when modalities have very different scales and when one is missing; dimensionality explodes.
• Late fusion (decision-level): train one model per modality, combine outputs by averaging, weighted voting, or a small meta-learner (stacking). Simple, robust to a missing modality, trains in parallel — but learns no cross-modal interaction.
• Intermediate / hybrid fusion: fuse at one or more hidden layers; the common deep-learning default.
• Attention-based fusion (cross-attention): queries come from one modality, keys and values from the other, so each text token attends to the relevant image regions. This is how BLIP, Flamingo-style and LLaVA-style models bind vision to language.
• Tensor / bilinear fusion: outer product of two feature vectors captures every pairwise multiplicative interaction; dimension is the product, so low-rank approximations (MCB, MFB, MUTAN) are used.
• Gated fusion: a learned gate per modality decides how much each contributes for that particular sample — useful when quality varies (blurred image, noisy audio).
• Model-agnostic versus model-based is the other textbook split: model-agnostic covers early/late fusion (any classifier); model-based covers kernel methods, graphical models and neural architectures designed for fusion.

Alignment and training objectives

• Explicit alignment is the task itself (align subtitles to audio); implicit alignment happens inside the model as a by-product of attention weights.
• Contrastive learning (CLIP-style): take N image–text pairs in a batch, form an N×N similarity matrix, and train so that the N matched pairs score high and the N²−N mismatched pairs score low, using a symmetric cross-entropy (InfoNCE) loss with a learnable temperature. Result: zero-shot classification by writing class names as text prompts.
• Masked multimodal modelling: mask text tokens and/or image patches and predict them using the other modality — forces grounding.
• Matching heads: image–text matching (binary: do these go together?) and word–region alignment.
• Generative objectives: caption generation (autoregressive language-model loss conditioned on image features), and text-conditioned diffusion for image generation.
• Instruction-tuned VLMs (LLaVA pattern), the current dominant recipe: frozen vision encoder → small trainable projector (linear layer or MLP) mapping visual features into the LLM's token embedding space → frozen or LoRA-tuned LLM. Cheap, because only the projector and adapters train.
• Co-learning / transfer: pretrain on abundant web image–text pairs, then fine-tune on a small labelled Indian-language or medical set. Modality distillation teaches a single-modality student using a multimodal teacher ("privileged information" at training time only).

Representative models and tasks

• CLIP / ALIGN / SigLIP — dual-encoder contrastive, built for retrieval and zero-shot recognition; cannot generate text.
• BLIP-2, Flamingo, LLaVA, Qwen-VL, Gemini, GPT-4-class VLMs — encoder + LLM, able to caption, answer questions, read charts and documents.
• Whisper — speech → text (speech recognition and translation); audio-visual speech recognition adds lip video for noisy conditions.
• Stable Diffusion / DALL·E-class — text → image: a text encoder conditions a denoising diffusion model through cross-attention.
• Indian-context models: AI4Bharat's IndicTrans/IndicWhisper line for Indian-language speech and text, and Sarvam/Krutrim multilingual multimodal assistants. Capability claims move fast — verify current versions before quoting them in an interview.
• Tasks to be able to name: image captioning, visual question answering (VQA), cross-modal retrieval, text-to-image generation, visual grounding/referring expressions, document VQA and OCR-based understanding, audio-visual event detection, video question answering, multimodal sentiment analysis, multimodal recommendation.
• Datasets/benchmarks: MS-COCO, Flickr30k, Conceptual Captions, LAION (image–text pairs), VQAv2, GQA, TextVQA, DocVQA, MSR-VTT (video), CMU-MOSEI (multimodal sentiment), MMMU and MMBench for modern VLM evaluation.

Limitations, evaluation and Indian regulatory context

• Missing modality at inference degrades or breaks joint models. Remedies: modality dropout during training, learned "missing" tokens, imputation from the available modality, or falling back to a unimodal head.
• Modality imbalance / collapse: the easier modality dominates and the model ignores the other (common in VQA, where answers are guessable from the question alone). Diagnose by ablation: test text-only and image-only baselines.
• Alignment is expensive: web pairs are noisy, captions are short and biased; Indian-language image–text pairs are scarce, so transliteration and machine translation are used, which injects errors.
• Compute: training a CLIP-scale model from scratch needs hundreds of GPU-days; the practical Indian path is fine-tuning open checkpoints or training only a projector.
• Hallucination and object hallucination: VLMs describe objects that are not in the image; measured by metrics such as POPE/CHAIR.
• Evaluation metrics: classification accuracy, VQA accuracy (consensus over 10 human answers), Recall@K and median rank for retrieval, BLEU/METEOR/ROUGE/CIDEr/SPICE for captioning, FID and CLIP-score for generation, WER for speech.
• Indian legal and standards context: the Digital Personal Data Protection Act, 2023 governs personal data — face images, voice and health scans are personal data requiring notice and consent, with higher duties for Significant Data Fiduciaries; its Rules were notified in 2025 and are being phased in, so check the current commencement position. Biometric multimodal systems touching Aadhaar fall under the Aadhaar Act, 2016 and UIDAI regulations; deepfake and synthetic-media obligations for intermediaries flow from the IT Act, 2000 and the IT Rules, 2021, with amendments on synthetic-content labelling under active notification — treat specifics as current-law-dependent. Medical imaging AI is regulated as software by CDSCO under the Medical Devices Rules, 2017. India also has IS/ISO/IEC 23053 and the ISO/IEC 42001 AI-management-system standard adopted through BIS, and NITI Aayog's Responsible AI principles as non-binding guidance.

Key points

• Modality - one channel of data (text, image, audio, video, sensor); multimodal AI relates two or more.
• Heterogeneity gap - modalities differ in structure and scale, so each needs its own encoder before comparison.
• Complementarity vs redundancy - the two reasons extra modalities help: new information, and robustness.
• Five challenges - representation, alignment, fusion, translation, co-learning.
• Joint representation - single shared space; needs all modalities at test time.
• Coordinated representation - separate spaces constrained to agree by similarity; enables retrieval and zero-shot.
• Early fusion - combine features before modelling; rich interaction, brittle to missing data.
• Late fusion - combine per-modality decisions; robust and parallel, but learns no interaction.
• Cross-attention - queries from one modality, keys/values from another; the standard modern fusion.
• Bilinear/tensor fusion - outer product captures all pairwise interactions; needs low-rank tricks.
• Contrastive InfoNCE loss - pulls matched pairs together, pushes N²−N mismatches apart, with learnable temperature.
• CLIP - dual encoder, zero-shot classification by text prompts, no text generation.
• Projector (LLaVA pattern) - small MLP mapping image features into LLM token space; cheapest way to build a VLM.
• Modality dropout - randomly drop a modality in training so the model survives it missing at inference.
• Modality collapse - model leans on the easy modality; detect by unimodal ablation baselines.
• Zero-shot transfer - classify unseen labels by embedding their names as text and comparing cosine similarity.
• Metrics - Recall@K (retrieval), CIDEr (captioning), VQA accuracy, FID (generation), WER (speech).
• DPDP Act, 2023 - face, voice and health data are personal data; consent and purpose limitation apply to multimodal datasets.

Formulas and examples

Example 1 — cosine similarity and zero-shot choice. Image embedding I = (0.6, 0.8, 0.0). Text embeddings: T₁ "a photo of a tractor" = (0.5, 0.5, 0.707); T₂ "a photo of a bus" = (0.0, 1.0, 0.0).

• ‖I‖ = √(0.36 + 0.64) = 1.0; ‖T₁‖ = √(0.25 + 0.25 + 0.4998) ≈ 1.0; ‖T₂‖ = 1.0.
• I·T₁ = 0.6(0.5) + 0.8(0.5) + 0 = 0.30 + 0.40 = 0.70.
• I·T₂ = 0 + 0.8(1.0) + 0 = 0.80.
• Both norms are 1, so similarities are 0.70 and 0.80 → the model predicts "bus".
• With τ = 0.1: logits 7.0 and 8.0; softmax → e⁷ = 1096.6, e⁸ = 2981.0; sum = 4077.6; P(bus) = 2981.0/4077.6 ≈ 0.731, P(tractor) ≈ 0.269.
• Lower the temperature to τ = 0.05: logits 14 and 16; e¹⁴ = 1.2026×10⁶, e¹⁶ = 8.8861×10⁶; P(bus) = 8.8861/(8.8861+1.2026) ≈ 0.881 — smaller τ sharpens the distribution.

Example 2 — contrastive loss for one batch row. Batch of N = 4. For image I₁ the similarities with the four captions are (matched first): 0.9, 0.2, 0.1, 0.0, with τ = 0.1 → logits 9, 2, 1, 0.

• e⁹ = 8103.1, e² = 7.389, e¹ = 2.718, e⁰ = 1.000; denominator = 8114.2.
• P(match) = 8103.1/8114.2 = 0.99863 → loss for this row = −ln(0.99863) ≈ 0.00137.
• If the encoders were untrained and all four similarities were equal, P = 1/4 and the row loss would be −ln(0.25) = 1.386 — so ln N is the random-chance loss, a standard sanity check.

Example 3 — fusion dimensions and parameter cost. Text feature 768-d (BERT CLS), image feature 2048-d (ResNet-50 pool), audio 128-d.

• Early fusion (concatenation): 768 + 2048 + 128 = 2,944-d. A following 512-unit layer needs 2,944 × 512 + 512 = 1,507,840 parameters.
• Bilinear fusion of text and image only: 768 × 2048 = 1,572,864 terms; a 512-way classifier on that would need over 805 million weights — hence low-rank factorisation.
• Late fusion: three 512-unit heads on 768, 2048 and 128 inputs = 768×512 + 2048×512 + 128×512 + 3×512 = 393,216 + 1,048,576 + 65,536 + 1,536 = 1,508,864 parameters, but each head trains independently and any one can run alone.

Example 4 — retrieval metrics. A cross-modal retrieval test on 1,000 image queries over 5,000 captions. The correct caption appears at rank 1 for 420 queries, within top 5 for 710, within top 10 for 820.

• R@1 = 420/1000 = 42.0%; R@5 = 71.0%; R@10 = 82.0%.
• Sum of recalls (rsum, a common single number) = 42 + 71 + 82 = 195.
• Random baseline R@1 = 1/5000 = 0.02%, so the model is about 2,100× better than chance at rank 1.

Example 5 — detecting modality collapse by ablation. A VQA model scores 68.0% with image + question. Question-only baseline scores 44.0%; image-only scores 24.0%.

• Gain of multimodal over the best unimodal = 68.0 − 44.0 = 24.0 percentage points, a relative gain of 24.0/44.0 = 54.5%.
• If the full model had scored 46%, the gain would be 2 points — evidence that the image is nearly ignored and the model is exploiting language priors.

Where it is used

• E-commerce (Flipkart, Myntra, Meesho, Amazon India): image + title + review fusion for catalogue cleanup, duplicate detection, attribute extraction and visual search ("find a kurta like this photo").
• Healthcare (Qure.ai, Niramai, Tata 1mg, Apollo): chest X-ray or CT plus the radiology text report; CDSCO-regulated as medical-device software, and patient images are personal data under the DPDP Act, 2023.
• Fintech and KYC (Paytm, PhonePe, banks under RBI's video-KYC rules): face image versus ID-document photo plus liveness audio/video; also multimodal document fraud checks on cheques and invoices.
• Indian-language assistants and accessibility (AI4Bharat, Sarvam, Jio, Bhashini): speech + text + screen-image understanding for 22 scheduled languages; OCR of Devanagari/Tamil/Malayalam signage with language models.
• Content moderation and media (Hotstar, ShareChat, Meta/Google India trust-and-safety teams): audio, video frames and caption text jointly scored; deepfake detection under IT Rules obligations.
• Industry and mobility (Ola, Tata Elxsi, Bosch, ISRO/remote sensing): camera + LiDAR + radar fusion for ADAS; satellite imagery plus textual records for crop and land-use mapping.
• Job roles: multimodal ML engineer, computer-vision engineer, applied scientist (vision-language), MLOps engineer for GPU inference, data annotation lead for paired datasets.

Interview questions

• Q: What makes a model multimodal rather than just multi-input? A: It must relate information *across* modality types that differ in structure, learning an aligned or joint representation — not merely accept two numeric feature blocks from the same kind of data.
• Q: Difference between joint and coordinated representations? A: Joint maps all modalities into one shared space and needs all of them at inference; coordinated keeps separate spaces tied by a similarity constraint, so each modality can be encoded independently and compared later — which is what makes retrieval scalable.
• Q: When would you choose late fusion over early fusion? A: When modalities are often missing, arrive at different times, or have very different noise levels, and when you want to train and deploy per-modality models independently.
• Q: Explain CLIP's training objective in one minute. A: For a batch of N image–text pairs it computes an N×N cosine-similarity matrix, scales by a learned temperature, and applies cross-entropy along both rows and columns so the N diagonal pairs win against all mismatched pairs.
• Q: Why can CLIP classify images it never saw labels for? A: Class names are turned into text prompts and embedded by the same aligned space, so classification becomes nearest-neighbour search between the image embedding and the candidate text embeddings.
• Q: What does the temperature τ do in a contrastive loss? A: It scales logits; a small τ sharpens the softmax and punishes hard negatives more, a large τ flattens it — too small destabilises training, which is why it is learned and clipped.
• Q: How does cross-attention bind text to an image? A: Text tokens act as queries while image patch embeddings supply keys and values, so each word aggregates the visual regions most relevant to it.
• Q: What would you do if 30% of your records have no audio? A: Train with modality dropout plus a learned missing-modality token, or use late fusion with a unimodal fallback head; never silently zero-fill, as zeros are a valid feature value the model will misread.
• Q: How do you prove the extra modality is actually helping? A: Run unimodal ablations and compare against the fused model on the same split; if the gain over the best unimodal baseline is small, the model has collapsed onto one modality.
• Q: Why is bilinear fusion rarely used at full rank? A: The outer product of a-dim and b-dim vectors has a×b terms, so parameters explode (768×2048 ≈ 1.57 million features before the classifier); low-rank factorisations such as MFB approximate it.
• Q: Why do vision-language models hallucinate objects? A: The language decoder is a strong prior and can continue a plausible sentence with weak visual grounding, especially when the projector compresses visual detail or the object is small or unusual.
• Q: Which Indian law governs a face-plus-voice dataset you collect for a KYC model? A: The Digital Personal Data Protection Act, 2023, with its 2025 Rules being phased in — face and voice are personal data needing notice, specific consent and purpose limitation; Aadhaar-linked biometrics additionally fall under the Aadhaar Act, 2016 and UIDAI regulations.

Common mistakes

• Zero-padding a missing modality and calling it imputation — the model cannot tell "absent" from "measured as zero" unless you add a mask or a missing token.
• Forgetting to L2-normalise embeddings before cosine similarity, then wondering why the contrastive loss will not converge.
• Comparing embeddings from two independently trained encoders; without a joint training objective the spaces are unrelated and similarity is meaningless.
• Reporting a multimodal accuracy with no unimodal baseline, so a language-prior model looks like genuine visual understanding.
• Letting in-batch negatives leak: duplicate or near-duplicate pairs in the same batch become false negatives and push apart items that genuinely match.
• Mixing fusion level with fusion mechanism in exams — "early/intermediate/late" is *where*, while "concatenation/bilinear/attention/gating" is *how*.
• Fine-tuning a huge VLM end-to-end on a few thousand Indian-language samples and overfitting, when training only the projector or a LoRA adapter is both cheaper and better.

Related topics

Semi-supervised and self-supervised learning · Transformers and attention mechanisms · Convolutional neural networks and vision architectures · Natural language processing and large language models · Transfer learning and fine-tuning (LoRA/PEFT) · Generative models: GANs, VAEs and diffusion