Computer vision basics

In short

Computer vision is the branch of artificial intelligence that makes a machine extract meaning from images and video — a 2-D or 3-D array of numbers in, a label, measurement or description out. It spans classical image processing (filtering, edges, thresholding, morphology, features) and learned representations (convolutional neural networks, vision transformers), and the two are still used together in industry. It is examined in AI&DS and CSE syllabi as "Digital Image Processing"/"Computer Vision", and in GATE DA/CS through convolution arithmetic, filtering and CNN parameter counts. In India it is applied at scale in UIDAI's Aadhaar face and fingerprint pipelines, FASTag/ANPR tolling on NHAI highways, DigiYatra airport boarding, UPI cheque and KYC document reading, and quality inspection lines at Tata, Bosch and Wipro PARI.

Explained

What an image is, and how it is represented

• A digital image is a sampled, quantised function f(x, y) giving intensity at spatial position (x, y). Sampling fixes resolution (how many pixels); quantisation fixes bit depth (how many levels per pixel).
• An 8-bit grayscale pixel holds 0–255 (0 = black, 255 = white). A colour image holds 3 channels, so shape is H × W × 3. OpenCV reads channels as BGR, PIL/Matplotlib as RGB — a classic bug source.
• Colour spaces: RGB (display, additive); HSV/HSL — hue, saturation, value — separates colour from brightness, so a saffron-flag or skin-tone threshold is far more robust in HSV; YCbCr/YUV separates luma from chroma and underlies JPEG and video compression; LAB is perceptually uniform for colour difference.
• Grayscale conversion is a weighted sum, not an average, because the eye is most sensitive to green: Y = 0.299R + 0.587G + 0.114B (ITU-R BT.601).
• Image types: binary (1 bit), grayscale, colour, multispectral/hyperspectral (ISRO Resourcesat, Cartosat data), depth or RGB-D (stereo, LiDAR, structured light), and video (a time sequence of frames).
• Neighbourhoods and connectivity: 4-connectivity (edge neighbours) and 8-connectivity (including diagonals) decide how blobs and contours are traced.

Point operations and histograms

• Point (intensity) transforms map each pixel independently: negative s = L−1−r; log transform s = c·log(1+r) expands dark detail; power-law/gamma s = c·r^γ (γ < 1 brightens, γ > 1 darkens) corrects display response.
• Contrast stretching linearly maps [r_min, r_max] to [0, 255].
• A histogram h(k) counts pixels of intensity k; normalised it is the probability p(k) = h(k)/(MN). It shows exposure and contrast but carries no spatial information — two very different images can share a histogram.
• Histogram equalisation uses the cumulative distribution as the transform, s_k = (L−1)·Σ p(j), spreading intensities to use the whole range. It is global and can amplify noise; CLAHE (contrast-limited adaptive HE) equalises small tiles with a clip limit and is standard for X-rays and low-light CCTV.
• Thresholding: global (fixed T), Otsu's method (choose T that maximises between-class variance — optimal for a bimodal histogram), and adaptive/local thresholding (T from a local mean or Gaussian window) for uneven illumination, as on a photographed answer sheet or scanned bill.

Spatial filtering and convolution

• Correlation vs convolution: convolution flips the kernel by 180° before the sliding sum; for symmetric kernels the two are identical, which is why deep-learning "convolution" layers are really correlations.
• Smoothing (low-pass): box/mean blur; Gaussian blur (separable, weights fall off with distance, standard denoiser); median filter — non-linear, order-statistic, the best cheap cure for salt-and-pepper noise because it does not average the outlier in; bilateral filter smooths while preserving edges by weighting on both distance and intensity difference.
• Sharpening (high-pass): Laplacian (second derivative, isotropic, zero-sum kernel); unsharp masking = original + k·(original − blurred).
• Edge detection finds intensity discontinuities. First-derivative operators: Sobel (3×3, with smoothing), Prewitt, Roberts. Gradient magnitude |∇f| = √(Gx² + Gy²), direction θ = atan2(Gy, Gx).
• Canny edge detector — the examinable 5-step pipeline: (1) Gaussian smoothing; (2) gradient magnitude and direction (Sobel); (3) non-maximum suppression to thin ridges to 1 px; (4) double thresholding into strong and weak edges; (5) hysteresis — keep weak edges only if connected to a strong one. Gives thin, connected, well-localised edges.
• Laplacian of Gaussian (LoG) / Difference of Gaussian (DoG) detect edges as zero-crossings and blobs across scale.
• Padding and stride: 'valid' (no pad, output shrinks), 'same' (pad so output size = input size for stride 1), stride s downsamples by ~s.
• Noise models: Gaussian (sensor/thermal), salt-and-pepper (impulse, bit errors), Poisson/shot (low light), speckle (multiplicative — SAR, ultrasound). Match the filter to the noise.

Morphology, geometry and classical features

• Morphological operations work on binary/grayscale shape using a structuring element: erosion shrinks foreground (removes specks), dilation grows it (closes gaps), opening = erosion then dilation (removes small objects, keeps size), closing = dilation then erosion (fills small holes), gradient = dilation − erosion (outline), top-hat/black-hat extract bright/dark details. Standard for cleaning up a thresholded ANPR number plate before OCR.
• Connected component labelling turns a binary mask into counted, measurable blobs (area, centroid, bounding box) — the basis of simple counting systems.
• Geometric transforms: translation, rotation, scaling, shear (affine, 6 degrees of freedom, preserves parallel lines, 3 point-pairs needed) and homography/projective (8 DoF, 4 point-pairs, maps a plane to a plane — used to "de-skew" a photographed document or a cricket pitch view).
• Interpolation on resampling: nearest-neighbour (blocky, keeps labels — use for masks), bilinear (default), bicubic/Lanczos (sharper, slower).
• Image pyramids (Gaussian and Laplacian) give multi-scale processing and scale-invariant search.
• Hough transform detects lines (ρ = x cosθ + y sinθ) and circles by voting in parameter space — lane and coin detection.
• Keypoint features: Harris corner detector (uses the second-moment matrix; corners have two large eigenvalues), SIFT (scale- and rotation-invariant, 128-D descriptor; patent expired, now free in OpenCV), SURF, ORB (FAST keypoints + BRIEF binary descriptor — fast, free, the practical choice on edge devices). Matching uses Euclidean or Hamming distance with a ratio test and RANSAC to reject outliers — basis of panorama stitching and image alignment.
• Global descriptors: HOG (histogram of oriented gradients, with linear SVM — the pre-deep-learning pedestrian detector), LBP for texture, colour histograms. Compact and interpretable, but hand-designed and brittle to clutter.

Learned vision: CNNs and beyond

• A convolutional neural network learns its filters instead of hand-coding them. Key ideas: local receptive fields, weight sharing (same kernel slides everywhere → translation equivariance and far fewer parameters than a dense layer), and hierarchy (edges → textures → parts → objects).
• Layers: convolution (+ bias) → non-linearity (ReLU) → optional pooling (max or average, downsamples, gives small-shift invariance) → repeated blocks → global average pooling → fully connected → softmax for class probabilities. Batch normalisation stabilises and speeds training; dropout regularises dense layers.
• Receptive field grows with depth, kernel size and stride; two stacked 3×3 convolutions see 5×5 with fewer parameters than one 5×5 — why 3×3 kernels dominate.
• Loss and training: cross-entropy loss for classification, optimised by SGD with momentum or Adam, with learning-rate schedules; mini-batches; early stopping on validation loss.
• Landmark architectures: LeNet-5 (digits), AlexNet (2012, ImageNet breakthrough), VGG (uniform 3×3 stacks), ResNet (skip/residual connections let 50–152 layers train by easing gradient flow), Inception (multi-scale branches), MobileNet/EfficientNet (depthwise-separable convolutions for phones), Vision Transformer (ViT) (splits an image into patch tokens and uses self-attention — needs large data or strong augmentation), and CLIP-style image–text models enabling zero-shot classification.
• Transfer learning is the normal industrial route: take ImageNet-pretrained weights, replace the classifier head, freeze early layers, fine-tune later ones — works with a few hundred labelled images per class instead of millions.
• Data augmentation: flips, random crops, rotation, scaling, colour jitter, blur, Cutout, Mixup. Never flip when orientation carries meaning (text, medical laterality, speedometer dials).
• Preprocessing discipline: resize, convert colour space, scale to [0, 1] and normalise with the *same* mean/std as the pretrained model; compute statistics on the training split only.
• Evaluation: accuracy, precision, recall, F1, confusion matrix, top-5 accuracy; macro-average for class imbalance. For pixel tasks, IoU/Dice. Split by *subject or source*, not by random frame, or near-duplicate frames leak between train and test and inflate scores.
• Limitations: needs large labelled data; sensitive to domain shift (trained on daylight, deployed at night), adversarial perturbations and spurious correlations; poor at reasoning about what it has not seen; compute- and energy-hungry.

Cameras, pipeline and the Indian compliance picture

• Pinhole camera model: a 3-D point projects as u = f_x·X/Z + c_x, v = f_y·Y/Z + c_y. Intrinsics (f_x, f_y, c_x, c_y, skew) sit in the matrix K; extrinsics are the rotation R and translation t of the camera in the world. Calibration (OpenCV chessboard method) also estimates radial/tangential distortion coefficients — mandatory before any measurement in mm.
• Depth: stereo disparity gives Z = f·B/d (B = baseline, d = disparity in pixels); structured light and ToF/LiDAR measure directly.
• End-to-end project pipeline: acquire (camera, lens, lighting — in factory vision, lighting fixes more problems than any algorithm) → preprocess → annotate → train/choose model → evaluate → optimise (quantise, prune) → deploy → monitor drift and retrain.
• Storage and compression: JPEG (lossy, DCT-based — avoid for annotation masks), PNG (lossless), WebP; raw sensor formats for scientific work. Repeated JPEG saving visibly destroys fine edges.
• India-specific practice: personal images and biometrics are personal data — biometrics were a "sensitive personal data" category under the IT Act's SPDI Rules, 2011, but the DPDP Act creates no separate sensitive class and applies one standard to all personal data; the Digital Personal Data Protection Act, 2023 governs their processing — notice, purpose limitation and consent, with rules and enforcement being rolled out, so check the current notification status before designing a face system. The Aadhaar Act, 2016 (as amended) and UIDAI specifications govern biometric capture devices and authentication. Information Technology Act, 2000 section 66E penalises capturing/publishing private images without consent. BIS standards exist for biometric image data interchange (the ISO/IEC 19794 series adopted as Indian Standards) and for CCTV equipment; quote the current IS/edition from the BIS catalogue, as numbers are revised. MeitY/STQC empanelment applies to biometric devices, and DigiYatra runs on a stated consent-and-deletion policy.

Key points

• Digital image - a sampled, quantised 2-D intensity function; resolution comes from sampling, bit depth from quantisation.
• Pixel ranges - 8-bit gives 0–255 and 2⁸ = 256 levels per channel; a colour pixel is 3 bytes.
• BGR vs RGB - OpenCV loads BGR; wrong order gives blue-looking faces and broken pretrained-model inputs.
• HSV - separates hue from brightness, so colour-based segmentation survives lighting change.
• Histogram - intensity distribution only, no spatial information; equalisation spreads it, CLAHE does it locally with a clip limit.
• Otsu's method - picks the global threshold maximising between-class variance; needs a bimodal histogram.
• Convolution - kernel flipped then slid; "same" padding keeps output size at stride 1.
• Median filter - non-linear, the right tool for salt-and-pepper noise; Gaussian is for additive Gaussian noise.
• Sobel - first-derivative edge operator with built-in smoothing; magnitude √(Gx²+Gy²), direction atan2(Gy,Gx).
• Canny - smooth, gradient, non-maximum suppression, double threshold, hysteresis; gives thin connected edges.
• Morphology - opening removes specks, closing fills holes; erosion shrinks, dilation grows the foreground.
• Affine vs homography - 6 DoF (3 point-pairs) vs 8 DoF (4 point-pairs); only homography handles perspective.
• Harris / SIFT / ORB - corner detector, scale-invariant descriptor, fast binary descriptor; RANSAC removes bad matches.
• CNN advantages - local receptive fields, weight sharing and pooling give translation tolerance with few parameters.
• ResNet skip connections - carry gradients past many layers, making very deep networks trainable.
• Transfer learning - pretrained backbone + new head is the default when labels are few.
• Camera calibration - intrinsics, extrinsics and distortion; required before converting pixels to millimetres.
• Stereo depth - Z = f·B/d; depth error grows with the square of distance.

Formulas and examples

1. Image size and storage cost. A 1920 × 1080 RGB frame at 8 bits per channel:

• Pixels = 1920 × 1080 = 2,073,600. Bytes = 2,073,600 × 3 = 6,220,800 ≈ 5.93 MiB (6.22 MB) uncompressed.
• At 25 fps for one hour: 6,220,800 × 25 × 3600 = 5.599 × 10¹¹ bytes ≈ 521 GiB raw — so a CCTV system must compress; H.264 at 4 Mbit/s stores the same hour in 4 × 10⁶ × 3600/8 = 1.8 × 10⁹ bytes ≈ 1.8 GB, about a 311× reduction (5.599 × 10¹¹ / 1.8 × 10⁹ — compare like units; dividing 521 GiB by 1.8 GB mixes GiB and GB).

2. Grayscale and gamma. A pixel is R = 200, G = 100, B = 50.

• Y = 0.299(200) + 0.587(100) + 0.114(50) = 59.8 + 58.7 + 5.7 = 124.2 → 124.
• Normalise: r = 124/255 = 0.4863. Apply γ = 0.5 (brighten) with c = 1: s = 0.4863^0.5 = 0.6974 → 0.6974 × 255 = 177.8 → 178. With γ = 2.2 it would be 0.4863^2.2 = 0.205 → 52, much darker.

3. Histogram equalisation on a 3-bit image. 64 pixels, L = 8, counts n_0…n_7 = 8, 10, 12, 14, 10, 6, 3, 1.

• Cumulative counts: 8, 18, 30, 44, 54, 60, 63, 64. CDF = that ÷ 64.
• s_k = round(7 × CDF): 7×0.125 = 0.875 → 1; 7×0.28125 = 1.969 → 2; 7×0.46875 = 3.281 → 3; 7×0.6875 = 4.813 → 5; 7×0.84375 = 5.906 → 6; 7×0.9375 = 6.563 → 7; 7×0.9844 = 6.891 → 7; 7×1 = 7.
• Mapping 0→1, 1→2, 2→3, 3→5, 4→6, 5→7, 6→7, 7→7: the crowded low levels are spread out and the sparse top levels merge — exactly the expected behaviour.

4. Sobel gradient at a pixel. Neighbourhood (rows top→bottom) = [[10, 10, 10], [10, 50, 90], [90, 90, 90]], with Gx kernel [[−1,0,1],[−2,0,2],[−1,0,1]] and Gy [[−1,−2,−1],[0,0,0],[1,2,1]].

• Gx = (−1·10 + 0 + 1·10) + (−2·10 + 0 + 2·90) + (−1·90 + 0 + 1·90) = 0 + 160 + 0 = 160.
• Gy = (−1·10 −2·10 −1·10) + 0 + (1·90 + 2·90 + 1·90) = −40 + 360 = 320.
• |∇f| = √(160² + 320²) = √(25,600 + 102,400) = √128,000 ≈ 357.8; θ = atan2(320, 160) = 63.4° — a strong, mainly horizontal edge (gradient points across it).

5. CNN output sizes and parameter count. Input 224 × 224 × 3. Conv1: 64 filters, 7 × 7, stride 2, padding 3; then max-pool 3 × 3, stride 2, padding 1.

• Conv1 output side = floor[(224 − 7 + 2×3)/2] + 1 = floor[223/2] + 1 = 111 + 1 = 112 → 112 × 112 × 64.
• Conv1 parameters = (7 × 7 × 3 + 1) × 64 = (147 + 1) × 64 = 9,472.
• Pool output = floor[(112 − 3 + 2)/2] + 1 = floor[111/2] + 1 = 55 + 1 = 56 → 56 × 56 × 64; pooling has 0 parameters.
• Compare a dense layer doing the same first mapping: 224×224×3 = 150,528 inputs to 112×112×64 = 802,816 outputs needs ≈ 1.2 × 10¹¹ weights — about 12.8 million times more than the 9,472, which is why convolution exists.

6. Stereo depth and error. f = 800 px, baseline B = 0.12 m, measured disparity d = 16 px.

• Z = 800 × 0.12/16 = 6.0 m. If disparity is mis-measured by 1 px (d = 17): Z = 96/17 = 5.65 m — a 35 cm error at 6 m. At d = 4 px, Z = 24 m and a 1-px error gives 96/5 = 19.2 m, a 4.8 m error: depth uncertainty grows as Z².

import cv2, numpy as np
img  = cv2.imread("plate.jpg")                 # BGR, uint8
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (5, 5), 1.4)     # kernel odd, sigma 1.4
edges = cv2.Canny(blur, 80, 160)               # low:high ≈ 1:2
_, bw = cv2.threshold(gray, 0, 255,
                      cv2.THRESH_BINARY + cv2.THRESH_OTSU)
k  = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
bw = cv2.morphologyEx(bw, cv2.MORPH_CLOSE, k)  # fill character gaps
n, lbl, stats, cent = cv2.connectedComponentsWithStats(bw)
print(n - 1, "blobs")                          # minus the background label

Where it is used

• Identity and public services: UIDAI Aadhaar face/fingerprint/iris quality checks and deduplication, DigiYatra face boarding at Delhi, Bengaluru and Hyderabad airports, passport and driving-licence photo validation.
• Transport and smart cities: ANPR for FASTag and e-challan enforcement on NHAI corridors, red-light and speed cameras, Bengaluru and Pune traffic command centres, Indian Railways coach and track inspection.
• Manufacturing and agriculture: surface and dimensional inspection at Tata Steel, Bosch, Hyundai and Wipro PARI lines; grain and produce grading; drone crop-health imagery (Kisan drones) and ISRO/Bhuvan satellite land-use mapping.
• Healthcare: chest X-ray and TB screening (Qure.ai), diabetic-retinopathy screening (Aravind Eye Hospital, Forus Health), oral and cervical cancer screening pilots in public health programmes.
• Banking, retail and media: cheque and KYC document OCR, video KYC liveness checks, Reliance and Flipkart warehouse barcode and damage detection, content moderation and sports analytics.
• Job roles: computer vision engineer, ML engineer, data scientist (vision), embedded/edge vision developer, annotation lead and MLOps engineer — hiring at TCS Research, Infosys, Wipro, Qure.ai, Niramai, Mad Street Den, Ola, Flipkart, Bosch, Samsung R&D and ISRO centres.

Interview questions

• Q: Why is grayscale a weighted sum rather than the mean of R, G and B? A: Human luminance sensitivity is highest for green and lowest for blue, so BT.601 weights 0.299/0.587/0.114 preserve perceived brightness; a plain mean distorts it.
• Q: When would you use a median filter instead of a Gaussian? A: For salt-and-pepper (impulse) noise — the median discards the outlier, whereas a Gaussian averages it in and smears the spike across the neighbourhood.
• Q: List the steps of the Canny edge detector. A: Gaussian smoothing, gradient magnitude and direction by Sobel, non-maximum suppression, double thresholding, and hysteresis linking of weak edges to strong ones.
• Q: What does histogram equalisation not fix? A: Anything spatial — blur, noise or local contrast; it only remaps intensities globally, and can amplify noise in flat regions. CLAHE handles the local case.
• Q: Why does a CNN need far fewer parameters than a fully connected network for images? A: Weight sharing — one small kernel is reused at every position — plus local connectivity, so parameters scale with kernel size and channels, not with image size.
• Q: What is a receptive field and why do we stack 3 × 3 convolutions? A: The input region influencing one output unit; two 3 × 3 layers cover 5 × 5 with 18C² weights against 25C² for a single 5 × 5, and add an extra non-linearity.
• Q: What does max pooling give up, and when is it wrong? A: It discards precise location for small-shift invariance; it is wrong when exact spatial detail matters, as in segmentation or precise measurement, where stride or dilated convolutions are preferred.
• Q: Difference between an affine transform and a homography? A: Affine has 6 DoF, preserves parallel lines and needs 3 point-pairs; a homography has 8 DoF, models perspective and needs 4 point-pairs on a plane.
• Q: Your model hits 99% in the lab and fails at the customer plant. What do you check first? A: Domain shift — lighting, camera, lens, working distance and product variants — plus data leakage from near-duplicate frames in the split; re-collect on-site data and fine-tune.
• Q: Why calibrate a camera before measuring an object? A: Lens distortion and unknown focal length/principal point make pixel distances non-linear in world units; calibration gives intrinsics and distortion coefficients so pixels convert correctly to millimetres.
• Q: How do you train a defect classifier with only 300 images? A: Transfer learning on an ImageNet-pretrained backbone with a new head, strong augmentation, class weighting or oversampling for rare defects, and cross-validation because the test set is tiny.
• Q: Is "convolution" in a CNN really convolution? A: Mathematically it is cross-correlation (no kernel flip); since the kernel is learned, the flip is irrelevant to the result.
• Q: What legal constraint applies before deploying face recognition in an Indian office? A: Facial images are personal data (biometric data) under the Digital Personal Data Protection Act, 2023, which has no separate "sensitive personal data" class — lawful notice and consent, purpose limitation, retention limits and security are required; these duties apply from 13 May 2027 under the DPDP Rules 2025; until then IT Act s.43A and the SPDI Rules 2011 (which treat biometrics as sensitive personal data) govern.

Common mistakes

• Forgetting OpenCV's BGR order, so colour thresholds and pretrained-model inputs are silently wrong.
• Using an even-sized or non-zero-sum kernel where the theory needs odd size (no defined centre) or zero sum (a Laplacian that brightens the image).
• Computing output size with the wrong padding convention, or omitting the bias when counting layer parameters.
• Normalising with statistics computed on the whole dataset — including the test split — which leaks information.
• Splitting video frames randomly, so near-identical neighbouring frames appear in both train and test and accuracy looks far better than it is.
• Flipping or rotating images in augmentation when orientation is semantic (text, dials, medical left/right).
• Treating a 403 or dark, blurred, low-resolution input as a model failure instead of fixing acquisition — lighting and lens choices beat algorithm tuning in factory vision.
• Reporting only accuracy on an imbalanced defect dataset where 98% of parts are good; precision, recall and the confusion matrix are what matter.

Related topics

Object detection and segmentation · Edge AI and model optimisation · Natural language processing basics · Text embeddings and semantic search · Speech recognition and audio AI · Time series forecasting