Edge AI and model optimisation

In short

Edge AI means running a trained model on or near the device that produces the data - a phone, camera, microcontroller, car ECU, POS terminal or factory gateway - instead of sending data to a cloud GPU. It matters because edge hardware has a fraction of the memory, compute and power of a server, so the model must be compressed and the runtime tuned: quantisation, pruning, distillation and graph optimisation typically cut size 4x and latency 2-4x with under 1% accuracy loss. Indian syllabuses examine it under AI/ML hardware, embedded AI and TinyML electives, with numericals on model size, FLOPs, memory bandwidth and latency budgets. It is a live hiring area at Qualcomm India, Samsung R&D, Bosch, Nvidia Bengaluru, Tata Elxsi, Niramai, Wadhwani AI, Ather and almost every Indian computer-vision startup, and is pushed by the DPDP Act, 2023 preference for keeping personal data local.

Explained

What edge AI is and why it is used

• Cloud inference: device sends raw data over the network, server runs the model, result comes back. Edge inference: model runs on the device; nothing or only a summary leaves it.
• Five drivers, in the order interviewers ask them: latency (no round trip - needed for driver monitoring, AR, robot control), bandwidth and cost (a 2 MP camera at 25 fps is ~2-8 Mbps of H.264/H.265 video, and over 1 Gbps if sent uncompressed; sending inference results is a few kbps), privacy/compliance (raw faces, voices and health data never leave the device - matches the DPDP Act, 2023 data-minimisation duty), reliability (works with no connectivity - rural India, mines, tunnels), energy (radio transmission often costs more joules than local compute).
• Tiers of "edge": on-sensor/MCU (TinyML, mW, kB of RAM) → mobile SoC (phone NPU, 1-10 W) → edge box/gateway (Jetson, Intel NUC, 10-60 W) → on-premise server (near-edge, 100 W+).
• Hybrid patterns: *cascade* (tiny always-on model wakes a bigger one - "Hey Siri" style keyword spotting), *split computing* (first layers on device, rest in cloud), *edge training is rare* - usually only fine-tuning or federated updates.
• Limits: fixed memory, no batching (batch size 1 hurts GPU utilisation), thermal throttling, no easy model rollback, fragmented hardware, and models must be re-optimised per chip.

Hardware for edge inference

• CPU (ARM Cortex-A): universal, uses NEON SIMD; good for INT8 with XNNPACK. Cortex-M with CMSIS-NN for MCU-class TinyML.
• GPU: mobile GPUs (Adreno, Mali) via OpenCL/Vulkan; Nvidia Jetson family (Nano, Orin Nano, AGX Orin) with CUDA and TensorRT.
• NPU / DSP / accelerators: Qualcomm Hexagon, Apple Neural Engine, Google Edge TPU (Coral), Intel VPU/NPU, MediaTek APU. Fixed-function, INT8/INT4-first, very high TOPS/W but support only a subset of operators - unsupported ops fall back to CPU and destroy latency.
• FPGA: reconfigurable, deterministic latency; used in industrial and defence vision.
• Metrics: TOPS/TFLOPS (peak ops per second), TOPS/W (efficiency), memory bandwidth (GB/s - often the real bottleneck), arithmetic intensity (FLOPs per byte moved). A model is *compute-bound* or *memory-bound*; roofline analysis decides which.
• Rule of thumb: a model is memory-bound when its arithmetic intensity is below the chip's FLOPs-per-byte ratio - true of depthwise convolutions, 1x1 convolutions and all LLM decoding.

Quantisation

• Idea: store and compute weights/activations in low precision instead of FP32. FP32 → INT8 gives 4x smaller model, 2-4x faster, far less energy (an INT8 multiply-accumulate costs roughly an order of magnitude less energy than FP32).
• Affine (asymmetric) quantisation: r = S(q − Z), where r = real value, q = integer, S = scale, Z = zero-point. Symmetric: Z = 0, used for weights.
• Granularity: per-tensor (one S for the whole tensor), per-channel/per-axis (one S per output channel - standard for weights, much better accuracy), per-group/block (e.g. 128 weights per scale - used in LLM 4-bit formats).
• Post-training quantisation (PTQ): quantise an already trained model. *Dynamic range* - weights INT8, activations quantised on the fly (easy, good for RNN/transformer). *Full integer (static)* - needs a calibration set of 100-500 representative samples to record activation ranges (min-max, percentile, KL-divergence/entropy, MSE methods).
• Quantisation-aware training (QAT): insert fake-quantise nodes during training/fine-tuning so the network learns to tolerate rounding; gradients pass through with the straight-through estimator (STE). Recovers most of the accuracy PTQ loses; costs a training run.
• Common precisions: FP32, FP16, BF16 (same exponent range as FP32, fewer mantissa bits - training-friendly), INT8 (the workhorse), INT4/NF4 and FP8 (mainly LLM weights), binary/ternary (research, big accuracy cost).
• Why accuracy drops: outlier activations (notably in transformer layers), depthwise convolutions with very different per-channel ranges, and batch-norm folding errors. Fixes: per-channel scales, clipping at a percentile, SmoothQuant-style scale migration, keeping sensitive layers (first conv, last classifier, softmax, layer-norm) in higher precision - mixed precision.
• Batch-norm folding: BN is merged into the preceding convolution's weights and bias before quantisation - must be done first, or scales are wrong.

Pruning, distillation and architecture choices

• Pruning: remove weights or structures. *Unstructured* (zero individual weights) - high sparsity, but needs sparse kernels to give real speedup, so often only shrinks the file. *Structured* (drop whole channels, filters, heads, layers) - gives real speedup on ordinary hardware. *Semi-structured* 2:4 sparsity is accelerated by Nvidia Ampere and later.
• Criteria: magnitude (|w|), L1/L2 norm of a filter, first-order (gradient × weight) saliency, Taylor expansion of the loss. Iterative prune-and-fine-tune beats one-shot pruning; typical schedule: prune 20%, fine-tune, repeat.
• Lottery-ticket observation: a sparse subnetwork found by pruning can be retrained from its original initialisation to match the dense model - cited in vivas.
• Knowledge distillation: a small *student* is trained to match a large *teacher*. Loss = α × CE(hard labels) + (1 − α) × T² × KL(student softmax at temperature T ‖ teacher softmax at T). Temperature T > 1 softens the distribution and carries "dark knowledge" about class similarity. Variants: response-based (logits), feature-based (intermediate activations), self-distillation.
• Low-rank factorisation: replace a weight matrix W (m×n) by AB with rank k, cutting parameters from mn to k(m+n); tensor decomposition (Tucker, CP) for convolution kernels.
• Efficient architectures (designed, not compressed): MobileNet v1-v3 (depthwise separable convolution, inverted residuals, h-swish), SqueezeNet, ShuffleNet, EfficientNet/EfficientNet-Lite (compound scaling), MobileViT and MobileNetV4 for mobile transformers, YOLO-Nano/YOLOv8n and SSD-MobileNet for detection, Whisper-tiny for speech.
• Depthwise separable convolution: a K×K depthwise convolution per channel followed by a 1×1 pointwise convolution; cost drops by roughly 1/K² + 1/N compared with a standard convolution (N = output channels) - about 8-9x for K = 3.
• Neural architecture search (NAS) / hardware-aware NAS: search architectures with measured on-device latency in the objective, not FLOPs.
• For LLMs at the edge: weight-only INT4 (GPTQ, AWQ, GGUF/llama.cpp k-quants), KV-cache quantisation, grouped-query attention, speculative decoding, LoRA adapters instead of full fine-tunes.

Runtime and graph optimisation

• Pipeline: train (PyTorch/TensorFlow) → export (ONNX, SavedModel, TorchScript) → convert and optimise (TFLite/LiteRT converter, TensorRT builder, OpenVINO Model Optimizer, Core ML tools, SNPE/QNN for Qualcomm) → deploy with a runtime → benchmark on the real device.
• Graph-level optimisations: constant folding, dead-node removal, operator fusion (Conv+BN+ReLU into one kernel - saves memory traffic, the biggest single win), layout transformation (NCHW ↔ NHWC), kernel auto-tuning, memory-arena planning and buffer reuse, weight sharing/clustering, FP16 or INT8 engine building.
• Runtimes: LiteRT (formerly TensorFlow Lite) and TFLite Micro, ONNX Runtime (with execution providers), TensorRT and TensorRT-LLM, OpenVINO, ExecuTorch, NCNN/MNN, llama.cpp, Apache TVM (compiler-based), MediaPipe for full pipelines.
• Delegates/execution providers route supported subgraphs to GPU/NPU and the rest to CPU; count the fallbacks - a single unsupported op in the middle causes costly tensor copies each frame.
• Always benchmark end to end: pre-processing (resize, colour conversion, normalisation), inference, post-processing (NMS, decoding) and copy time. Report p50 and p95/p99 latency, not just the mean, and measure after warm-up with a locked clock to separate thermal throttling.
• Deployment concerns: model signing and integrity, OTA model updates, A/B rollout, on-device monitoring for drift, and model theft (an on-device model is extractable - weight encryption and TEE only raise the cost).

Measurement, trade-offs and Indian context

• Accuracy-latency-size-power is a four-way trade-off; the deliverable is a *Pareto curve*, and the chosen point follows the product's latency budget (e.g. 33 ms per frame for 30 fps, including everything else the CPU must do).
• FLOPs are a proxy, not a measure - two models with equal FLOPs can differ 3x in latency because of memory access cost, operator support and parallelism.
• Energy per inference (mJ) decides battery life: E = average power × latency. Measure with an inline power monitor, not from the datasheet.
• Standards and regulation in India: the Digital Personal Data Protection Act, 2023 (rules notified in 2025, obligations phasing in) makes on-device processing attractive because data minimisation and reduced transfer lower the compliance surface; a Data Fiduciary's duties still apply to whatever is collected. Bharat Smart Vision / BIS work on CCTV cybersecurity - the Essential Requirements for CCTV under MeitY's notification (effective 2025) require security testing for surveillance cameras sold in India, which pushes analytics on-camera. Electronics manufacturing is supported by the India Semiconductor Mission and the PLI schemes; AI hardware procurement by government follows GIGW and CERT-In directions. Specific IS numbers for AI systems are still emerging - the IS/ISO/IEC 22989 and 23053 adoptions cover AI terminology and ML framework; quote subject rather than number if unsure.
• Cost framing: a Jetson Orin Nano developer kit is roughly ₹40,000-55,000 and a Raspberry Pi 5 with an accelerator a few thousand rupees, against recurring cloud GPU inference billed per hour - the break-even for a 24x7 camera is usually a few months. Prices vary; check current listings.

Key points

• Edge AI - running inference on or near the data source instead of in the cloud; chosen for latency, bandwidth, privacy, offline working and energy.
• Quantisation - mapping FP32 values to low-precision integers via r = S(q − Z); INT8 gives ~4x size reduction and 2-4x speedup.
• PTQ vs QAT - PTQ needs only a calibration set and minutes; QAT needs retraining but recovers accuracy on hard models.
• Per-channel quantisation - separate scale per output channel; essential for depthwise convolutions and MobileNets.
• Calibration set - 100-500 unlabelled representative samples used to fix activation ranges; must match deployment data.
• Batch-norm folding - merge BN into the preceding convolution before quantising, else scales and accuracy are wrong.
• Unstructured pruning - zeroes weights, shrinks storage, needs sparse kernels for real speed; structured pruning removes channels and speeds up any hardware.
• 2:4 sparsity - two of every four weights zero; hardware-accelerated on Nvidia Ampere and later.
• Knowledge distillation - small student trained on the teacher's soft outputs at temperature T; KD loss scaled by T².
• Depthwise separable convolution - depthwise + pointwise; roughly 8-9x cheaper than a 3×3 standard convolution.
• Operator fusion - combining Conv+BN+activation into one kernel to avoid writing intermediate tensors to memory.
• Delegate / execution provider - runtime plug-in that offloads a subgraph to GPU, DSP or NPU; unsupported ops fall back to CPU.
• Memory-bound vs compute-bound - arithmetic intensity (FLOPs/byte) versus the chip's ratio decides which; most edge models are memory-bound.
• TOPS/W - efficiency metric for edge accelerators; peak TOPS is rarely achieved in practice.
• Latency budget - 30 fps means 33 ms total for capture, pre-processing, inference and post-processing, not 33 ms of inference.
• p95/p99 latency - what the user feels; always report with p50 and after warm-up to expose thermal throttling.
• FLOPs are a proxy - measure on-device latency; equal-FLOP models can differ several-fold.
• TinyML - ML on microcontrollers in kB of RAM with TFLite Micro/CMSIS-NN; keyword spotting, anomaly detection, person detection.

Formulas and examples

Example 1 - INT8 quantisation parameters and round-trip error. A layer's activations after calibration lie in [−4.0, 12.0]. Quantise to asymmetric UINT8 (0-255).

• S = (12.0 − (−4.0))/255 = 16.0/255 = 0.062745 per step.
• Z = round(−rmin/S) = round(4.0/0.062745) = round(63.75) = 64.
• Encode r = 5.0: q = round(5.0/0.062745) + 64 = round(79.69) + 64 = 80 + 64 = 144.
• Decode: r' = S(q − Z) = 0.062745 × (144 − 64) = 0.062745 × 80 = 5.0196.
• Quantisation error = 0.0196, i.e. 0.39% of the value and well within the half-step bound S/2 = 0.0314.

Example 2 - Model size, memory bandwidth and the real bottleneck. A 25.6-million-parameter ResNet-50 is deployed on a board with 25 GB/s memory bandwidth and 1.3 TFLOPS of FP16 compute. The model needs 8.2 GFLOPs per 224×224 image.

• FP32 size = 25.6 × 10⁶ × 4 bytes = 102.4 MB. INT8 size = 25.6 MB → 4x compression.
• Weight traffic per inference at INT8 = 25.6 MB → time to read weights alone = 25.6/25,000 s = 1.024 ms.
• Compute time at peak = 8.2 × 10⁹ / 1.3 × 10¹² = 6.3 ms; achieving 40% of peak → ~15.8 ms.
• So this network is compute-bound here (6.3 ms ≫ 1.024 ms); quantisation helps mainly through faster INT8 arithmetic, not bandwidth.
• At INT8 with a 3x effective arithmetic gain, inference ≈ 15.8/3 ≈ 5.3 ms → ~190 fps headline, before pre/post-processing.

Example 3 - Depthwise separable convolution saving. Layer: input 56×56×128, 3×3 convolution, 256 output channels, stride 1, same padding (output 56×56×256).

• Standard MACs = 3² × 128 × 256 × 56 × 56 = 9 × 128 × 256 × 3,136 = 924,844,032 MACs (≈0.92 GMAC, 1.85 GFLOPs).
• Separable MACs = (9 × 128 + 128 × 256) × 3,136 = (1,152 + 32,768) × 3,136 = 33,920 × 3,136 = 106,373,120 MACs (≈0.106 GMAC).
• Ratio = 106,373,120/924,844,032 = 0.115 → 8.69x cheaper; check with formula 1/Cout + 1/K² = 1/256 + 1/9 = 0.0039 + 0.1111 = 0.115. ✔
• Parameters: standard = 9 × 128 × 256 = 294,912; separable = 9 × 128 + 128 × 256 = 1,152 + 32,768 = 33,920 (8.69x fewer).

Example 4 - Latency budget for an Indian retail camera. A shelf-monitoring camera must process 15 fps. Measured per frame: capture+resize 4 ms, normalise 2 ms, inference 48 ms (FP32 on CPU), NMS+decode 6 ms.

• Budget = 1000/15 = 66.7 ms; current total = 4 + 2 + 48 + 6 = 60 ms → 16.7 fps, just inside, with no margin for thermal throttling.
• Apply INT8 PTQ (inference 48 → 17 ms) and fuse resize+normalise into one pre-processing kernel (6 → 3 ms): total = 3 + 17 + 6 = 26 ms → 38.5 fps.
• With the Amdahl formula on inference alone: f = 48/60 = 0.8, k = 48/17 = 2.82 → speedup = 1/(0.2 + 0.8/2.82) = 1/(0.2 + 0.2837) = 2.07x, i.e. 60 → 29 ms; the extra gain came from the pre-processing fix.
• Energy: at 4.5 W average, E = 4.5 × 26 ms = 117 mJ per frame against 4.5 × 60 = 270 mJ before - 57% less.

Example 5 - Pruning plus quantisation for a TinyML microcontroller. A 1.2-million-parameter keyword-spotting model must fit 256 kB of flash on a Cortex-M4.

• FP32 size = 1.2 × 10⁶ × 4 = 4.8 MB - 18.75x too large.
• Structured pruning to 70% sparsity (s = 0.70) → 1.2 × 10⁶ × 0.30 = 360,000 parameters.
• INT8 quantisation → 360,000 × 1 byte = 360 kB - still over budget.
• Add INT4 weight quantisation for the two largest dense layers (60% of weights): 0.6 × 360,000 × 0.5 + 0.4 × 360,000 × 1 = 108,000 + 144,000 = 252 kB → fits, with ~4 kB spare before activation buffers.
• Activation arena must also fit RAM: peak two-layer working set 48 kB at INT8 against 128 kB SRAM - acceptable. Reported accuracy after QAT fine-tuning: 94.1% against 95.3% dense FP32, a 1.2-point loss.

# TFLite/LiteRT full-integer post-training quantisation with a calibration generator
import tensorflow as tf, numpy as np

def rep_data():                      # 100-500 real samples, same pre-processing as deployment
    for img in calib_images[:300]:
        yield [np.expand_dims(img.astype(np.float32), 0)]

conv = tf.lite.TFLiteConverter.from_saved_model("saved_model/")
conv.optimizations = [tf.lite.Optimize.DEFAULT]
conv.representative_dataset = rep_data
conv.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
conv.inference_input_type  = tf.int8   # fully integer in/out -> no float ops on the MCU
conv.inference_output_type = tf.int8
open("model_int8.tflite", "wb").write(conv.convert())

# PyTorch structured pruning + export, then measure on-device (never trust FLOPs alone)
import torch, torch.nn.utils.prune as prune
for m in model.modules():
    if isinstance(m, torch.nn.Conv2d):
        prune.ln_structured(m, name="weight", amount=0.3, n=2, dim=0)  # drop 30% of filters
        prune.remove(m, "weight")          # make the sparsity permanent before export
# fine-tune here, then: torch.onnx.export(...) and benchmark p50/p95 after warm-up

Where it is used

• Smartphones and consumer devices: Qualcomm India and Samsung R&D Bengaluru/Noida teams port camera denoising, face unlock, scene detection and on-device speech to Hexagon/NPU with INT8 and INT4 models.
• Video analytics and smart cities: on-camera ANPR, helmet and seatbelt detection, crowd counting for police and municipal projects (Bengaluru, Hyderabad, Pune traffic systems); MeitY's CCTV security requirements favour analytics inside the camera.
• Automotive and ADAS: Bosch, Continental, Tata Elxsi, KPIT and Ather run driver-monitoring and lane/object detection on automotive SoCs with hard latency budgets; validated for functional safety alongside ISO 26262-style processes.
• Industrial and agri IoT: vibration-based anomaly detection on motors and pumps (TinyML on Cortex-M), crop-disease detection on phones for farmers, grain-quality grading at mandis - Wadhwani AI and several agri-tech startups.
• Healthcare devices: Niramai thermal breast screening, portable ECG and retinal screening boxes in PHCs where connectivity is poor and patient data must stay local under the DPDP Act.
• Job roles: Edge AI / embedded ML engineer, model optimisation engineer, inference performance engineer, TinyML developer, AI SoC software engineer, MLOps engineer for on-device deployment.

Interview questions

• Q: Why quantise to INT8 rather than just use a smaller model? A: Quantisation keeps the trained architecture and accuracy almost intact while cutting memory 4x and using integer units that are far faster and more energy-efficient; a smaller model loses capacity. In practice both are combined.
• Q: What is the difference between PTQ and QAT, and when would you choose each? A: PTQ quantises a trained model using a calibration set only; QAT fine-tunes with simulated quantisation so the weights adapt. Start with PTQ - move to QAT only if the accuracy drop exceeds the budget, typically for MobileNet-style depthwise networks or very low bit widths.
• Q: What is a calibration dataset and what goes wrong if it is unrepresentative? A: A few hundred unlabelled samples used to estimate activation min/max ranges. If it misses the deployment distribution, scales clip real activations or waste range, and accuracy collapses on exactly the inputs you care about.
• Q: Explain zero-point and scale. A: Scale is the real-world size of one integer step; zero-point is the integer that represents exact zero, so that padded zeros and ReLU outputs stay exact. Weights use symmetric quantisation (Z = 0); activations usually asymmetric.
• Q: Why does unstructured pruning often give no speedup? A: Scattered zeros still occupy dense tensors and dense GEMM kernels, so compute is unchanged; you need sparse kernels or hardware support (for example 2:4 sparsity) to realise it. Structured pruning removes whole channels and always speeds up.
• Q: How does knowledge distillation transfer more information than training on labels? A: The teacher's softened probability distribution shows relative similarity between classes - a dog image scoring 0.7 dog, 0.2 wolf, 0.001 truck - which is richer supervision than the one-hot label.
• Q: Your model's FLOPs dropped 5x but latency only 1.3x. Why? A: The model is likely memory-bound or uses operators the runtime does not accelerate - depthwise convolutions, odd channel counts, or unsupported ops falling back to CPU with tensor copies. Profile per-layer and count delegate fallbacks.
• Q: What is operator fusion and why is it valuable? A: Merging consecutive ops such as Conv+BN+ReLU into a single kernel so intermediate tensors are never written to DRAM; it cuts memory traffic and kernel-launch overhead, usually the largest single graph-level win.
• Q: How do you decide between edge and cloud inference for an Indian product? A: Weigh latency requirement, connectivity reliability, per-device bandwidth cost, DPDP-driven privacy exposure, and whether the model fits the device budget; cameras and audio usually go edge, large LLM workloads usually stay in the cloud with a small on-device gate.
• Q: How do you run an LLM on an edge device? A: Use a small model (1-4 B parameters), weight-only INT4 with group-wise scales (GPTQ/AWQ/GGUF), quantise the KV cache, and accept that decoding is memory-bandwidth-bound so tokens/second tracks bandwidth divided by model bytes.
• Q: What should you report when benchmarking? A: End-to-end p50 and p95/p99 latency after warm-up, throughput at batch 1, peak RAM, model size, accuracy on the deployment set and energy per inference - all on the target device at a controlled temperature.
• Q: Why keep the first and last layers in higher precision? A: The first convolution sees raw input with wide dynamic range and few parameters, and the final classifier's logits directly decide the output; quantising them saves little size but costs disproportionate accuracy.
• Q: Can a model on a device be stolen? A: Yes - weights are present in the binary and can be extracted, so protection means encryption at rest, loading inside a TEE, and watermarking; it raises the attacker's cost rather than eliminating the risk.

Common mistakes

• Quoting FLOPs or parameter count as the performance result instead of measured on-device latency and memory on the target chip.
• Quantising before folding batch-norm, or using a single per-tensor scale for depthwise convolutions - both cause large, confusing accuracy drops.
• Using training data straight from the augmentation pipeline (or only 5-10 images) as the calibration set, so activation ranges do not match deployment.
• Reporting inference time only, then missing the frame-rate target because resize, normalisation, NMS and memory copies were never counted.
• Benchmarking the first few runs, or on a cold device, and reporting an average - hiding both warm-up cost and thermal throttling; report p95.
• Claiming a speedup from unstructured pruning with no sparse kernel, or reporting compressed file size as if it were runtime memory.
• Assuming an NPU runs the whole graph; one unsupported operator silently falls back to CPU and can make the "accelerated" model slower than plain CPU.
• Mixing units and bases - MB vs MiB, bits vs bytes, MACs vs FLOPs (1 MAC ≈ 2 FLOPs) - in size and throughput calculations.

Related topics

Computer vision basics · Object detection and segmentation · Speech recognition and audio AI · Large language models and prompt engineering · Embedded systems and microcontrollers · MLOps and model deployment