Image and video processing basics

In short

Image processing treats a picture as a two-dimensional signal f(x, y) of light intensity, and video adds a third dimension — time — so that redundancy between successive frames can be exploited. The subject sits at the meeting point of signals, mathematics and computing: sampling and quantization, filtering, transforms, segmentation, compression and codecs. In Indian universities it is a core or elective paper in the ECE/EEE syllabus, and GATE EC/IN sets questions on 2D convolution, histogram operations, masks and entropy-based compression. Employers hiring for these skills include ISRO and NRSC (satellite imagery), DRDO, Samsung R&D, Qualcomm, Ittiam, Nvidia, Bosch, Tata Elxsi, KLA and L&T Technology Services, plus every camera-ISP, ADAS, medical-imaging and CCTV-analytics team in the country.

Explained

Image formation, sampling and quantization

• A monochrome image is f(x, y) = i(x, y) · r(x, y): illumination i (incident light, 0 to ∞) times reflectance r (0 to 1 for opaque bodies, transmissivity for transparent).
• Digitisation = sampling + quantization. Sampling discretises the spatial coordinates (how many pixels), quantization discretises the amplitude (how many grey levels).
• An M × N image with L = 2^k grey levels needs M · N · k bits. Typical k = 8 → 256 levels, 0 = black, 255 = white.
• Spatial resolution = pixels per unit distance (dpi/ppi); intensity resolution = number of bits per pixel. Too few samples → *checkerboard / blocking*; too few levels → false contouring (visible bands in smooth gradients).
• 2D sampling theorem: sample at more than twice the highest spatial frequency in each direction, else aliasing appears as moiré patterns and jagged edges. Anti-aliasing = low-pass filter the optical image (an optical low-pass filter in front of the sensor, or lens blur) before sampling.
• Pixel neighbourhoods: N₄ (4 edge neighbours), N_D (4 diagonal), N₈ (all eight). Connectivity is 4-, 8- or m-connectivity (m- removes ambiguous diagonal double paths).
• Distance measures: Euclidean D_e, city-block D₄ (diamond), chessboard D₈ (square).
• Image file formats: BMP (raw), PNG (lossless, DEFLATE), JPEG (lossy DCT), TIFF, GIF; video containers MP4/MKV/TS carry coded streams.

Colour models and colour processing

• RGB — additive, used by sensors and displays; a 24-bit RGB image has 16.7 million colours. CMY/CMYK — subtractive, for printing: C = 1 − R, etc.
• HSI / HSV — hue, saturation, intensity/value; decouples colour from brightness, so grey-level algorithms can be applied to I alone. Natural for human description and for colour-based segmentation.
• YCbCr — luma Y and two chroma differences; used by JPEG, MPEG and all broadcast. Y = 0.299R + 0.587G + 0.114B (ITU-R BT.601); BT.709 for HD uses 0.2126/0.7152/0.0722.
• Chroma subsampling exploits the eye's lower chroma acuity: 4:4:4 (no subsampling), 4:2:2 (horizontal half), 4:2:0 (half in both directions — the broadcast and streaming default, 50% of raw data saved).
• A single-chip camera uses a Bayer colour filter array (RGGB); demosaicing interpolates the two missing channels at each pixel.

Intensity transformations and spatial filtering

• Point operations (per pixel): negative s = L − 1 − r; log s = c·log(1 + r) expands dark values (used for Fourier-spectrum display); power-law/gamma s = c·r^γ (γ < 1 brightens, γ > 1 darkens — gamma correction of monitors, γ ≈ 2.2); contrast stretching; thresholding; bit-plane slicing.
• Histogram h(r_k) = n_k, number of pixels with level r_k; the normalised histogram p(r_k) = n_k/MN is the PDF estimate. A dark image's histogram crowds to the left, a low-contrast one is narrow.
• Histogram equalization applies the transformation s_k = (L − 1)·Σ p(r_j), the CDF, giving an approximately uniform histogram and maximum global contrast. Discrete levels mean the result is only approximately flat, and it may amplify noise. Histogram specification (matching) shapes the histogram to a desired one; CLAHE equalises in tiles with contrast clipping (standard in medical and X-ray imaging).
• Spatial filtering slides an m × n mask; correlation multiplies and sums as-is — g(x, y) = Σ Σ w(s, t)·f(x + s, y + t) — while convolution first rotates the mask by 180°, giving f(x − s, y − t). Padding (zero, replicate, mirror) handles borders.
• Smoothing (low-pass): box/averaging mask (all 1/9 in 3 × 3) and Gaussian; blur noise but also edges. Median filter (order-statistic, non-linear) removes salt-and-pepper noise while preserving edges — the classic exam answer. Also min, max, midpoint, alpha-trimmed filters.
• Sharpening (high-pass): based on derivatives. Laplacian ∇²f = f(x+1,y) + f(x−1,y) + f(x,y+1) + f(x,y−1) − 4f(x,y); sharpened image g = f − ∇²f (for a centre coefficient of −4). Unsharp masking g = f + k(f − f̄); k > 1 is high-boost filtering.
• Separability: a Gaussian or averaging mask of size n × n splits into two 1D passes, cutting cost from O(n²) to O(2n) per pixel.
• Noise models: Gaussian (sensor/thermal), Rayleigh, Gamma, exponential, uniform, impulse (salt-and-pepper, from bit errors), speckle (multiplicative, in SAR and ultrasound), periodic (from interference — removed by a notch filter in the frequency domain).

Frequency-domain processing and transforms

• The 2D DFT converts the image to frequency; low frequencies = slow brightness variation (background), high frequencies = edges, noise and fine detail. Displayed after shifting DC to the centre and log-scaling.
• Filtering steps: pad to size P × Q (P ≥ 2M − 1, Q ≥ 2N − 1 to avoid wrap-around error) → multiply by (−1)^x+y to centre → DFT → multiply by H(u, v) → inverse DFT → take real part → uncentre → crop.
• Ideal LPF (sharp cut-off D₀) causes ringing (Gibbs); Butterworth of order n gives a smooth transition with little ringing; Gaussian gives no ringing at all. The high-pass versions are H_HP = 1 − H_LP.
• Convolution theorem: spatial convolution ↔ frequency multiplication, so large masks are cheaper via FFT.
• DCT (Discrete Cosine Transform) has excellent energy compaction for natural images and uses only real arithmetic — hence JPEG and all mainstream video codecs use an 8 × 8 (or 4 × 4 integer) DCT rather than the DFT.
• Other transforms examined: Walsh–Hadamard (only ±1, fast), Haar and Discrete Wavelet Transform — multi-resolution, gives LL/LH/HL/HH subbands, used in JPEG 2000 and for denoising; KLT (Hotelling) is optimal in energy compaction but image-dependent, so not practical.

Segmentation, edges and morphology

• Segmentation partitions an image into regions. Approaches: discontinuity-based (point, line, edge) and similarity-based (thresholding, region growing, region splitting and merging, watershed, clustering such as k-means).
• Edge detection masks: Roberts (2 × 2), Prewitt and Sobel (3 × 3 first-derivative; Sobel weights the centre row/column by 2, so it smooths while differentiating), Laplacian (second derivative — detects edges by zero crossings but is noise-sensitive), LoG / Marr–Hildreth (Gaussian smoothing then Laplacian, zero crossings).
• Canny edge detector (the most-asked algorithm): 1) Gaussian smoothing; 2) gradient magnitude and angle; 3) non-maximum suppression to thin edges to one pixel; 4) double thresholding (T_high, T_low) and edge linking by hysteresis. Optimal in detection, localisation and single response.
• Thresholding: global (fixed T), Otsu's method (chooses T maximising between-class variance, equivalently minimising within-class variance — automatic and bimodal-optimal), adaptive/local thresholding for uneven illumination.
• Hough transform detects lines and circles by voting in parameter space; the normal form ρ = x·cosθ + y·sinθ avoids infinite slope.
• Morphology on binary images with a structuring element B: erosion A ⊖ B shrinks/removes small objects; dilation A ⊕ B grows and fills gaps; opening (erode then dilate) removes specks and smooths contours; closing (dilate then erode) fills small holes; boundary = A − (A ⊖ B); plus hit-or-miss, thinning, thickening, skeletonisation.
• Features and recognition: region properties (area, perimeter, centroid, compactness, Euler number), chain codes, moments and Hu's seven invariant moments (invariant to translation, rotation, scale), texture by GLCM, descriptors SIFT/SURF/ORB; modern pipelines use CNNs (classification, detection with YOLO/SSD, segmentation with U-Net).

Image compression

• Redundancies removed: *coding* redundancy (unequal symbol probabilities), *spatial/interpixel* redundancy (neighbouring pixels are similar), *psychovisual* redundancy (detail the eye cannot see) and, in video, *temporal* redundancy.
• Lossless: Huffman, arithmetic, run-length (RLE), LZW, predictive DPCM lossless, bit-plane coding. Typical ratio 2:1 to 3:1; used for medical and legal images.
• Lossy: transform coding (DCT, wavelet) with quantization — ratios of 10:1 to 50:1 at visually acceptable quality.
• JPEG baseline steps: RGB → YCbCr → 4:2:0 subsampling → split into 8 × 8 blocks → level shift (subtract 128) → 2D DCT → divide by the quantization table (quality factor scales it) → zigzag scan → DC coded differentially (DPCM), AC run-length coded → Huffman coding. Decoder reverses this. Blocking artefacts appear at high compression.
• Fidelity measures: RMSE, PSNR in dB (above ~35 dB is usually good), SSIM (structural similarity — closer to perception), and subjective MOS on a 1–5 scale.

Video fundamentals and coding

• A video is a sequence of frames at a frame rate of 24, 25 (PAL/India), 30, 50 or 60 fps. India's analogue legacy was PAL-B/G, 625 lines, 25 fps; digital terrestrial TV from Doordarshan uses DVB-T2 with MPEG-4/H.264, and DTH/cable use MPEG-4 and HEVC.
• Interlaced vs progressive: interlaced (1080i) sends two fields of alternate lines per frame to halve bandwidth; progressive (720p, 1080p, 2160p/4K) sends full frames. Deinterlacing is needed to show interlaced content on flat panels.
• Resolutions: SD 720 × 576, HD 1280 × 720, Full HD 1920 × 1080, 4K UHD 3840 × 2160.
• Temporal redundancy removal — motion estimation: the current frame is split into macroblocks (16 × 16 in MPEG/H.264); each is matched in a reference frame within a search window; the displacement is the motion vector and the difference is the residual, which is then DCT-coded. Matching criteria: SAD (cheapest, used in hardware), SSD/MSE, MAD.
• Search algorithms: full/exhaustive search (best quality, most costly), three-step search, diamond search, logarithmic and hierarchical search.
• Frame types (GOP): I-frame — intra-coded, independent, random-access point; P-frame — predicted from a previous frame; B-frame — bidirectionally predicted from past and future frames (best compression, adds delay). A GOP such as IBBPBBPBBP repeats typically every 12–15 frames.
• Standards timeline: H.261 (video conferencing) → MPEG-1 (VCD) → MPEG-2/H.262 (DVD, DTH, digital TV) → MPEG-4 Part 2 → H.264/AVC (still the most widely deployed) → H.265/HEVC (~50% bitrate saving over H.264, CTUs up to 64 × 64) → H.266/VVC and the royalty-free AV1, both increasingly used in streaming. Codec deployment varies by platform and year, so check the current profile list before quoting one.
• Rate control chooses CBR (constant bitrate, for broadcast links) or VBR (variable, for storage/streaming); adaptive bitrate streaming (HLS, MPEG-DASH) switches between pre-encoded renditions — what JioHotstar and SonyLIV do over Indian mobile networks.
• Video quality and processing tasks: denoising, stabilisation, frame-rate conversion, super-resolution, scene-change detection, object tracking (Kalman filter, optical flow by Lucas–Kanade or Horn–Schunck).

Key points

• Image - a 2D function f(x, y) of intensity; digital image = sampled in space, quantized in amplitude.
• Storage size - M × N × k bits for greyscale; × 3 for colour before compression.
• Aliasing - under-sampling below the Nyquist rate; shows as moiré and jagged edges.
• False contouring - visible bands caused by too few grey levels (small k).
• Histogram equalization - CDF-based mapping that spreads intensities for maximum global contrast; can amplify noise.
• Convolution vs correlation - convolution flips the mask by 180°; identical for symmetric masks.
• Median filter - non-linear, the best cheap cure for salt-and-pepper noise; preserves edges.
• Laplacian - isotropic second derivative for sharpening; noise-sensitive, so smooth first (LoG).
• Sobel - first-derivative mask with built-in smoothing; gradient magnitude ≈ |Gx| + |Gy|.
• Canny - smooth, gradient, non-maximum suppression, hysteresis thresholding — the optimal edge detector.
• Otsu - automatic threshold maximising between-class variance for a bimodal histogram.
• Opening vs closing - opening removes small bright specks; closing fills small dark holes.
• DCT - real-valued, high energy compaction; the engine of JPEG and video codecs.
• Ringing - oscillations near edges from an ideal (sharp) frequency filter; avoided with Gaussian or Butterworth.
• YCbCr + 4:2:0 - separates luma from chroma and halves chroma in both directions, exploiting the eye.
• Motion estimation - macroblock matching gives a motion vector plus a residual; removes temporal redundancy.
• I, P, B frames - independent, forward-predicted and bidirectionally predicted; B gives the best compression and the most delay.
• PSNR - 10·log₁₀(MAX²/MSE) in dB; a convenient but imperfect proxy for perceived quality.

Formulas and examples

Example 1 — Storage and compression ratio. A colour image is 1024 × 768 with 8 bits per channel.

• Raw size = 1024 × 768 × 3 × 8 = 18,874,368 bits = 2,359,296 bytes = 2.25 MiB.
• Stored as a JPEG of 150 KiB (153,600 bytes): CR = 2,359,296 / 153,600 = 15.36 : 1.
• Redundancy removed R = 1 − 1/15.36 = 0.935, i.e. 93.5% of the data was redundant.

Example 2 — Histogram equalization (3-bit image, L = 8, 64 pixels).

• Check: 8 + 10 + 16 + 14 + 10 + 4 + 2 = 64 pixels. ✔ (CDF at r = 2 is (8+10+16)/64 = 34/64 = 0.531; 7 × 0.531 = 3.72 → 4.)
• New histogram: level 1 → 8 px, 2 → 10, 4 → 16, 5 → 14, 6 → 10, 7 → 4 + 2 = 6. Levels 0 and 3 are now empty — the discrete result is only approximately uniform, and levels merge at the top.

Example 3 — Sobel gradient at a pixel. The 3 × 3 neighbourhood is

with Gx (vertical-edge mask) = [−1 0 1; −2 0 2; −1 0 1] and Gy (horizontal-edge mask) = [−1 −2 −1; 0 0 0; 1 2 1].

• Gx = (−10 + 10) + (−20 + 20) + (−90 + 90) = 0 (no horizontal intensity change).
• Gy = −(10 + 20 + 10) + (90 + 180 + 90) = −40 + 360 = 320.
• Magnitude = √(0² + 320²) = 320; direction θ = tan⁻¹(320/0) = 90°, i.e. the gradient points vertically across a horizontal edge. Clipped to 8 bits the response saturates at 255 — so gradient images are usually scaled.

Example 4 — PSNR. An 8-bit image is compressed and the mean squared error between original and reconstruction is 25.

• RMSE = √25 = 5 grey levels.
• PSNR = 10·log₁₀(255² / 25) = 10·log₁₀(65,025 / 25) = 10·log₁₀(2601) = 10 × 3.4152 = 34.15 dB — borderline-good quality.
• If MSE improves to 9, PSNR = 10·log₁₀(65,025/9) = 10·log₁₀(7225) = 38.59 dB.

Example 5 — Video bitrate and the need for compression. Full HD 1920 × 1080, 4:2:0 at 8 bits, 25 fps.

• Bits per pixel for 4:2:0 = 8 (Y) + 8/4 + 8/4 = 12 bpp.
• Raw rate = 1920 × 1080 × 12 × 25 = 622,080,000 bits/s ≈ 622 Mbit/s.
• A one-hour film raw = 622.08 × 3600 / 8 ≈ 280 GB — impossible to stream.
• An H.264 stream at 5 Mbit/s gives CR = 622.08 / 5 ≈ 124 : 1; HEVC at ~2.5 Mbit/s roughly doubles that saving.

Where it is used

• Space and remote sensing: ISRO/NRSC (Bhuvan portal, Cartosat and Resourcesat imagery) for crop estimation, urban mapping and disaster assessment; SAR speckle filtering for RISAT data.
• Consumer camera and mobile ISP teams: Samsung R&D Bengaluru/Noida, Qualcomm, Nvidia and Ittiam work on demosaicing, denoising, HDR fusion, auto-focus/exposure and on-device codecs.
• Automotive and industrial vision: Bosch, Continental, KPIT, Tata Elxsi and L&T Technology Services build ADAS lane/pedestrian detection; KLA and machine-vision firms do wafer and PCB defect inspection.
• Medical imaging: X-ray, CT, MRI and ultrasound enhancement and segmentation at Siemens Healthineers, Philips, GE Healthcare and Indian startups working on TB and diabetic-retinopathy screening.
• Surveillance, biometrics and government systems: Aadhaar fingerprint and iris matching, ANPR for FASTag and traffic enforcement, face recognition in smart-city command centres — governed in India by the DPDP Act, 2023 and the IT Act, 2000 for handling personal and biometric data.
• Broadcast and OTT engineering: Doordarshan's DVB-T2 chain, DTH operators and OTT platforms doing transcoding, adaptive bitrate packaging and quality monitoring; typical job titles are image/video algorithm engineer, codec engineer, computer-vision engineer and ISP validation engineer.

Interview questions

• Q: What is the difference between sampling and quantization? A: Sampling discretises the spatial coordinates (how many pixels are taken), while quantization discretises the intensity amplitude (how many grey levels each pixel can take).
• Q: Why is the median filter preferred over an averaging filter for salt-and-pepper noise? A: Impulse pixels are extreme outliers; the median ignores them because it picks the middle rank, whereas an average is pulled towards the outlier and also blurs edges.
• Q: What does histogram equalization do and when does it fail? A: It maps intensities through the normalised CDF to spread them over the full range for maximum global contrast; it fails on images that are already well balanced or contain large uniform areas, where it over-amplifies noise — CLAHE is used instead.
• Q: Why does JPEG use the DCT rather than the DFT? A: The DCT is real-valued (no complex arithmetic), has better energy compaction for natural images, and its implicit even symmetry avoids the block-boundary discontinuities that the DFT would introduce.
• Q: Explain the steps of the Canny edge detector. A: Gaussian smoothing, gradient magnitude and direction, non-maximum suppression to thin edges, then hysteresis thresholding with a high and a low threshold to link edges while rejecting noise.
• Q: What is the difference between opening and closing? A: Opening is erosion followed by dilation and removes small bright objects and thin protrusions; closing is dilation followed by erosion and fills small holes and narrow gaps. Both preserve overall object size.
• Q: Why does an ideal low-pass filter produce ringing? A: Its sharp cut-off in frequency corresponds to a sinc function in space, whose side lobes create oscillations near edges — the Gibbs phenomenon; Butterworth and Gaussian filters taper the cut-off and avoid it.
• Q: What are I, P and B frames? A: I-frames are coded independently and allow random access; P-frames are predicted from earlier frames; B-frames are predicted from both earlier and later frames, giving the highest compression but adding encoding delay and reordering.
• Q: How does motion estimation reduce the bitrate? A: Each macroblock is matched to a similar block in a reference frame, so only a small motion vector and the low-energy residual need to be coded instead of the whole block.
• Q: Why is 4:2:0 chroma subsampling visually acceptable? A: Human vision has far lower spatial acuity for colour than for brightness, so halving chroma resolution in both directions saves half the data with little perceived loss.
• Q: Is a high PSNR always a good image? A: No — PSNR measures pixel-wise error only. A slightly shifted or mildly blurred image can score well while looking wrong, so SSIM or subjective MOS is used alongside it.
• Q: Which is cheaper for a 15 × 15 Gaussian blur — spatial convolution or FFT? A: Either separate the Gaussian into two 1D passes (30 multiplies per pixel instead of 225) or use the FFT; for masks beyond roughly 7 × 7 non-separable, frequency-domain filtering wins.

Common mistakes

• Confusing correlation with convolution — forgetting to rotate the mask by 180°, which only matters for asymmetric masks.
• Treating histogram equalization as producing a perfectly flat histogram; with discrete levels it only approximates one, and levels merge.
• Forgetting to normalise or scale filter outputs, so sharpened or gradient images saturate or go negative when stored as 8-bit.
• Mixing bits with bytes, and KB with KiB, in storage and bitrate calculations — a frequent source of answers off by a factor of eight.
• Not padding before DFT-based filtering, producing wrap-around error at the image borders.
• Claiming lossy compression is reversible, or quoting a compression ratio without saying at what quality or PSNR.
• Confusing frame rate with bitrate, or interlaced 1080i with progressive 1080p, when discussing video standards.

Related topics

Digital signal processing: filters and FFT · Machine learning and computer vision with CNNs · Embedded systems and DSP processors · Information theory and source coding · Pattern recognition · Multimedia communication and streaming protocols