Machine vision

In short

Machine vision is the use of cameras, lighting, optics and image-processing algorithms to let a machine or robot measure, inspect, locate and identify objects automatically. In robotics it closes the loop between sensing and motion — the camera tells the robot *where* the part is, so the robot need not rely on fixtures holding every part in an exact pose. It is examined in Mechatronics and Robotics papers, in GATE (Instrumentation, EC and the Mechanical "mechatronics/measurement" sections) and in automation electives, usually through camera geometry, image-processing steps, resolution calculations and system block diagrams. Indian employers — Bosch, Tata Motors, Maruti Suzuki, Hyundai, Ather, Cognex/Keyence India partners, Addverb, GreyOrange, Omron, ISRO and the semiconductor and pharma packaging industry — hire for vision inspection and vision-guided robotics roles.

Explained

What a machine vision system contains

• Scene and part: the object to be inspected or located, with its possible pose variation, surface finish and colour range defined up front.
• Illumination: the single most decisive choice; it creates the contrast the algorithm needs. Types: diffuse/dome (hides texture, good for shiny parts), ring, backlight (silhouette — best for dimension measurement), dark-field (scratches glow), coaxial, structured light (laser line for 3D), strobed LED (freezes motion).
• Lens/optics: focal length f, aperture (f-number), working distance, depth of field; telecentric lens gives no perspective error, so edges do not shift with height — standard for precision gauging.
• Image sensor: CCD (uniform, low noise, legacy) or CMOS (dominant now, faster, global or rolling shutter). Global shutter is mandatory for moving parts; rolling shutter skews them.
• Camera type: area-scan (2D matrix) for discrete parts; line-scan for continuous web (paper, textile, steel strip); 3D (stereo, time-of-flight, laser triangulation, structured light); smart camera with on-board processing.
• Interface and processing: GigE Vision / USB3 Vision / Camera Link to a PC or embedded board; GenICam is the common software interface standard. Processing in OpenCV, HALCON, Cognex VisionPro, LabVIEW Vision or NI/Keyence toolkits.
• Output and interfacing: digital I/O, PLC fieldbus (PROFINET, EtherCAT, Modbus TCP) or a pose sent to the robot controller.
• Classification by task: gauging (dimensions), flaw detection, identification (OCR/barcode), guidance (pose for a robot), counting/sorting, and verification (presence/absence).

Image fundamentals

• Digital image: a 2D array f(x, y) of pixels; spatial resolution = number of pixels (e.g. 1,280 × 1,024 ≈ 1.3 MP); intensity resolution = bit depth (8-bit → 256 grey levels).
• Colour models: RGB (sensor-native), HSV/HSI (hue separates colour from brightness — robust to illumination change, so used for colour sorting), grayscale (most inspection), Bayer filter → demosaicing produces colour from a monochrome sensor.
• Sampling and quantisation: continuous optical image → discrete pixels (sampling) and discrete grey levels (quantisation); under-sampling causes aliasing (moiré on fine grids).
• Histogram: plot of pixel count against grey level; shows exposure, contrast and whether a global threshold will separate object from background (bimodal histogram).

Image-processing pipeline (standard exam sequence)

1. Acquisition: trigger (encoder or photo-sensor), expose, transfer; fix gain, exposure and lighting so they never float.
2. Pre-processing (enhancement/restoration): noise removal — mean/Gaussian filter (blurs edges), median filter (best for salt-and-pepper, preserves edges); contrast stretching, histogram equalisation; shading/flat-field correction; geometric distortion correction.
3. Segmentation: separate object from background — global thresholding, Otsu's method (automatic threshold maximising between-class variance), adaptive/local thresholding (for uneven lighting), region growing, watershed, edge-based segmentation, colour segmentation in HSV.
4. Edge detection: gradient operators Sobel, Prewitt, Roberts; Laplacian (second derivative, zero crossings); Laplacian of Gaussian (LoG); Canny — the optimal multi-stage detector: Gaussian smoothing → gradient magnitude and direction → non-maximum suppression → hysteresis double thresholding.
5. Morphological operations (on binary images, with a structuring element): erosion shrinks and removes specks; dilation grows and fills pinholes; opening = erosion then dilation (removes noise, keeps size); closing = dilation then erosion (closes gaps); hit-or-miss, thinning/skeletonisation, boundary extraction, blob/connected-component labelling.
6. Feature extraction: area, perimeter, centroid, bounding box, orientation from second moments, circularity 4πA/P², aspect ratio, Euler number (holes), Hu moments (invariant to translation, rotation, scale), Hough transform (lines and circles from edge points), corner detectors (Harris), keypoint descriptors (SIFT/SURF/ORB — ORB is free of patent issues and standard in OpenCV).
7. Recognition/decision: template matching (normalised cross-correlation), geometric pattern matching (rotation/scale-tolerant), classical classifiers (k-NN, SVM), and CNN-based deep learning for cosmetic defects that cannot be described by rules.
8. Measurement and output: convert pixel results to millimetres via calibration, compare against tolerance, issue PASS/FAIL or a pose to the robot, log the image.

Camera model, calibration and 3D

• Pinhole model: a point in the camera frame (Xc, Yc, Zc) projects to image coordinates u = f·Xc/Zc, v = f·Yc/Zc; written in homogeneous form as s·[u v 1]ᵀ = K·[R | t]·[X Y Z 1]ᵀ.
• Intrinsic matrix K holds fx, fy (focal length in pixels), (cx, cy) principal point and skew — properties of the camera. Extrinsics [R | t] give the camera's pose relative to the world — 6 DOF.
• Lens distortion: radial (barrel/pincushion, coefficients k1, k2, k3) and tangential (p1, p2); corrected using the Brown–Conrady model before any measurement.
• Calibration methods: Zhang's method — photograph a planar chequerboard in 10–20 orientations, solve homographies, then refine all parameters by non-linear least squares (the calibrateCamera routine in OpenCV); also Tsai's method and direct linear transform (DLT) for the 3×4 projection matrix.
• Homography H (3×3, 8 DOF): maps one plane to another; 4 point correspondences give H, used for perspective correction of a flat part and for pixel-to-mm mapping on a conveyor.
• Stereo vision: two cameras with baseline b; disparity d = xL − xR; depth Z = f·b/d. Needs rectification (epipolar lines made horizontal) and correspondence matching (block matching, semi-global matching). Depth error grows with Z².
• Other 3D methods: laser triangulation (profile sensors on welds), structured light (projected pattern), time-of-flight (Azure Kinect-class sensors; note the Intel RealSense D400 series is active stereo, not ToF), shape-from-focus, photometric stereo.
• Hand-eye calibration: finds the fixed transform between camera frame and robot flange (eye-in-hand) or robot base (eye-to-hand) by solving AX = XB from several robot poses of a known target — without it a perfectly calibrated camera still makes the robot miss.

Vision-guided robotics and visual servoing

• Look-then-move (static): camera takes one image, computes part pose, robot moves open-loop to the computed point. Simple, used in most pick-and-place; accuracy limited by calibration.
• Visual servoing (closed loop): image feedback drives the robot continuously.
  • Position-based (PBVS/3D): pose is reconstructed from the image and the error is computed in Cartesian space — straight-line motion, sensitive to calibration errors.
  • Image-based (IBVS/2D): error computed directly on image features, related to robot velocity by the image Jacobian (interaction matrix) L: ṡ = L·v; control law v = −λ·L⁺·(s − s*). Robust to calibration error but the 3D path can be odd and may hit singularities or joint limits.
  • Hybrid (2½D) mixes both.
• Bin picking: 3D sensing + pose estimation + collision-free grasp planning; the hardest common industrial vision problem (reflective, overlapping, tangled parts).
• Other robot uses: seam tracking in welding, deburring path correction, conveyor tracking (encoder-synchronised), inspection on the arm, and AGV/AMR navigation by visual SLAM and fiducial markers (ArUco/AprilTag).

Performance, selection and limitations

• Resolution budget: choose pixel size so the smallest defect spans at least 2–3 pixels (sub-pixel edge fitting can reach ~1/5 pixel on good contrast).
• Accuracy vs repeatability: repeatability is usually 5–10× better than absolute accuracy; state measurement uncertainty, not just pixel count.
• Speed: cycle time = trigger + exposure + transfer + processing + output; line-rate limits for line-scan; frame rate falls as resolution rises.
• Advantages: non-contact, 100% inspection at line speed, objective and traceable, records images for audit, enables flexible feeding (no hard tooling per variant).
• Limitations: ambient-light sensitivity, specular and transparent surfaces, occlusion, colour metamerism, cost of good optics/lighting, need for engineering effort and retraining when the product changes, and the "black-box" explainability problem with deep models.
• Indian standards and practice: instrument accuracy claims trace through NABL-accredited calibration to national standards at CSIR-NPL; legal metrology aspects of weighing/measuring instruments fall under the Legal Metrology Act, 2009 and its rules. Dimensional tolerance and geometrical tolerancing of the features being gauged follow the ISO GPS family adopted as Indian Standards (ISO 2768 general tolerances; ISO 1101 geometrical tolerancing). Laser-based 3D sensors must meet laser product safety classification (IEC 60825 series, adopted in India); machine safety for the guarded cell follows IEC/ISO machine-safety standards, and workplace duties now sit under the Occupational Safety, Health and Working Conditions Code, 2020, one of the four Labour Codes in force from 21 November 2025. Electrical installation of the cell follows the CEA Safety Regulations and IS 732 for wiring practice. Exact IS numbers and editions change — verify the current edition on the BIS portal before quoting one in a specification.

Key points

• Machine vision - automatic acquisition and analysis of images for measurement, inspection, identification and robot guidance.
• Lighting first - contrast created optically costs nothing in software; backlight for size, dark-field for scratches, dome for shiny parts.
• Telecentric lens - parallel rays, no perspective error, essential for accurate dimensional gauging.
• Global shutter - captures all rows at once; rolling shutter distorts moving objects.
• Line-scan camera - builds the image one row at a time from a moving web; needs encoder synchronisation.
• Pipeline - acquire → pre-process → segment → extract features → recognise/measure → decide and output.
• Median filter - removes salt-and-pepper noise while keeping edges sharp, unlike mean/Gaussian.
• Otsu's method - picks the global threshold that maximises between-class variance of the histogram.
• Canny - smoothing, gradient, non-maximum suppression, hysteresis thresholding: the standard "good" edge detector.
• Opening vs closing - opening removes small specks; closing fills small holes and joins gaps.
• Blob analysis - labels connected regions and reports area, centroid, orientation and circularity.
• Hough transform - detects lines and circles by voting in parameter space; robust to gaps and noise.
• Intrinsics vs extrinsics - K describes the camera; [R|t] describes where it is; distortion coefficients correct the lens.
• Zhang calibration - multiple views of a planar chequerboard give intrinsics, extrinsics and distortion.
• Homography - 3×3, 8 DOF plane-to-plane map from 4 correspondences; gives pixel-to-mm on a flat plane.
• Stereo depth - Z = f·b/d; larger baseline or focal length improves depth resolution, error grows as Z².
• Hand-eye calibration - solves AX = XB for the camera-to-robot transform; without it the robot misses the target.
• IBVS vs PBVS - image-based control is robust to calibration error; position-based gives cleaner Cartesian paths.
• Sub-pixel edge fitting - interpolating the intensity profile across an edge gives roughly 1/5 to 1/10 pixel repeatability.

Formulas and examples

Example 1 — Lens and resolution selection for a gasket inspection. Field of view required 120 mm × 90 mm, camera 2,448 × 2,048 px, sensor 2/3″ (8.45 mm × 7.07 mm), working distance fixed at 300 mm, smallest defect 0.3 mm.

• Focal length: f ≈ (8.45 × 300)/120 = 21.1 mm → choose a standard 25 mm lens and re-check: FOV = 8.45 × 300/25 = 101 mm (slightly tight) — so a 16 mm lens gives FOV = 8.45 × 300/16 = 158 mm, comfortably covering 120 mm with margin. Pick 16 mm, or keep 25 mm and increase WD to 25 × 120/8.45 ≈ 355 mm.
• Pixel resolution with the 120 mm FOV across 2,448 px: R = 120/2,448 = 0.049 mm/pixel.
• Defect coverage: 0.3/0.049 ≈ 6.1 pixels — above the 2–3 pixel minimum, so the camera is adequate.
• Required resolution check: N = (120/0.3) × 3 = 1,200 px minimum; 2,448 px available.

Example 2 — Motion blur and strobe timing. Parts move on a conveyor at 0.5 m/s = 500 mm/s; allowed blur is half a pixel at 0.049 mm/pixel = 0.0245 mm.

• t_exp ≤ 0.0245/500 = 4.9 × 10⁻⁵ s = 49 µs.
• So a strobed LED of ~50 µs (or a short electronic shutter with very bright light) is needed; at 10 ms exposure the blur would be 500 × 0.01 = 5 mm — the part would be unreadable.

Example 3 — Stereo depth and its uncertainty. Stereo pair: f = 1,200 px, baseline b = 80 mm, measured disparity d = 48 px, disparity matching accuracy Δd = 0.25 px.

• Z = 1,200 × 80/48 = 2,000 mm = 2.0 m.
• ΔZ ≈ (Z²/(f·b))·Δd = (2,000²/(1,200 × 80)) × 0.25 = (4,000,000/96,000) × 0.25 = 41.67 × 0.25 ≈ 10.4 mm.
• At Z = 1.0 m: ΔZ ≈ (1,000,000/96,000) × 0.25 ≈ 2.6 mm — four times better, confirming the Z² law. Doubling the baseline to 160 mm halves the error to ≈ 5.2 mm at 2 m.

Example 4 — Pixel-to-world calibration and a robot pick. A fixed overhead camera views a flat conveyor. Two known marks 200.0 mm apart appear 3,968 px apart.

• Scale = 200.0/3,968 = 0.05040 mm/pixel.
• A part centroid is found at (1,420, 860) px; the calibrated origin (world 0,0) is at pixel (224, 180), with image x aligned to world X.
• ΔX = (1,420 − 224) × 0.05040 = 1,196 × 0.05040 = 60.28 mm; ΔY = (860 − 180) × 0.05040 = 680 × 0.05040 = 34.27 mm.
• Blob orientation from second moments is 18.5°, so the robot is commanded to (60.28, 34.27, z_pick) with tool rotation 18.5°. A 1-pixel centroid error costs only 0.05 mm, but a 1% scale error at 60 mm costs 0.6 mm — calibration dominates the error budget.

Example 5 — Throughput check. Exposure 0.05 ms, image transfer of a 5 MP mono image over GigE at 100 MB/s effective ≈ 5,000,000/100,000,000 s = 50 ms, processing 25 ms, I/O 2 ms.

• Cycle ≈ 0.05 + 50 + 25 + 2 ≈ 77 ms → about 13 parts/s maximum, i.e. ~46,800 parts/hour if the line can feed them.
• Needed: 20 parts/s (50 ms). The transfer is the bottleneck → move to 10 GigE/USB3, use a smart camera that processes on board, or reduce the region of interest.

# Classical pipeline: locate a part on a conveyor and report its pose in mm
import cv2, numpy as np
img  = cv2.imread("part.png", cv2.IMREAD_GRAYSCALE)
img  = cv2.medianBlur(img, 5)                       # salt-and-pepper noise
_, bw = cv2.threshold(img, 0, 255,
                      cv2.THRESH_BINARY + cv2.THRESH_OTSU)   # auto threshold
k    = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
bw   = cv2.morphologyEx(bw, cv2.MORPH_OPEN, k)      # remove specks
cnts, _ = cv2.findContours(bw, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
c    = max(cnts, key=cv2.contourArea)               # largest blob = the part
M    = cv2.moments(c)
cx, cy = M["m10"]/M["m00"], M["m01"]/M["m00"]       # centroid in pixels
(_, _), (w, h), theta = cv2.minAreaRect(c)          # orientation in degrees
A, P = cv2.contourArea(c), cv2.arcLength(c, True)
circ = 4*np.pi*A/(P*P)                              # circularity, 1 = circle
SCALE = 0.05040                                     # mm per pixel, calibrated
print(round(cx*SCALE, 2), round(cy*SCALE, 2), round(theta, 1), round(circ, 3))

Where it is used

• Automotive manufacturing: Maruti Suzuki, Tata Motors, Hyundai, Bosch and their Tier-1 suppliers use vision for weld-nut presence, bead inspection, gap-and-flush measurement, VIN/OCR traceability and vision-guided windshield and seat loading.
• Electronics and semiconductors: SMT solder-paste inspection and AOI on PCB lines (Dixon, Foxconn India, Syrma SGS), wire-bond and die inspection in the new ATMP/fab investments under the India Semiconductor Mission.
• Pharma and FMCG packaging: blister and vial inspection, fill-level and cap checks, label and batch-code verification (Sun Pharma, Cipla, Hindustan Unilever lines) — driven by Schedule M GMP and track-and-trace/barcode requirements for exports.
• Warehousing and logistics robotics: Addverb, GreyOrange, Rapyuta Robotics and e-commerce fulfilment centres use barcode/dimensioning vision, AMR navigation with fiducials and visual SLAM, and vision-based depalletising.
• Textiles, steel and paper: line-scan web inspection for weaving defects and surface flaws in rolling mills (Tata Steel, JSW).
• Agritech, space and defence: fruit grading and sorting machines, ISRO's satellite imaging and star-tracker/camera-based navigation, DRDO target recognition; job roles include Vision Engineer, Automation/Controls Engineer, ML-CV Engineer, Quality/Metrology Engineer and Robotics Integration Engineer.

Interview questions

• Q: Why is lighting considered more important than the algorithm in machine vision? A: Lighting creates the contrast between the feature and its background; if that contrast is absent in the raw image, no algorithm can reliably recover it, whereas good lighting makes even simple thresholding robust.
• Q: What is the difference between intrinsic and extrinsic camera parameters? A: Intrinsics (fx, fy, cx, cy, skew, plus distortion coefficients) describe the camera and lens themselves; extrinsics (R and t) describe the camera's position and orientation relative to the world or robot frame.
• Q: How is a camera calibrated in practice? A: By Zhang's method — capturing 10–20 images of a planar chequerboard in different orientations, detecting corner correspondences, estimating homographies for initial intrinsics, then refining intrinsics, extrinsics and distortion by non-linear least-squares minimisation of reprojection error.
• Q: What is hand-eye calibration and why is it needed? A: It determines the rigid transform between the camera frame and the robot flange or base by solving AX = XB from several robot poses viewing a fixed target; without it, correct image coordinates still map to the wrong robot coordinates.
• Q: Compare image-based and position-based visual servoing. A: IBVS computes error directly on image features using the image Jacobian and is robust to calibration error but can produce awkward 3D paths; PBVS reconstructs the 3D pose and gives straight Cartesian motion but is sensitive to calibration and model errors.
• Q: Why would you use a median filter instead of a Gaussian filter? A: The median removes impulsive salt-and-pepper noise without smearing edges, because it replaces a pixel with a neighbourhood median rather than a weighted average.
• Q: What are the stages of the Canny edge detector? A: Gaussian smoothing, gradient magnitude and direction computation, non-maximum suppression to thin edges, and hysteresis thresholding with two thresholds to keep connected weak edges.
• Q: When is a telecentric lens necessary? A: When measuring dimensions of parts with appreciable thickness or varying height, because a telecentric lens has no perspective magnification change, so apparent size does not vary with distance.
• Q: How does stereo depth accuracy vary with distance, and what can you do about it? A: Depth error grows roughly with the square of range (ΔZ ≈ Z²Δd/fb), so you increase the baseline or focal length, improve sub-pixel disparity matching, or switch to an active 3D sensor.
• Q: What is the difference between opening and closing? A: Opening (erode then dilate) removes small bright specks and thin protrusions while roughly preserving object size; closing (dilate then erode) fills small holes and bridges narrow gaps.
• Q: When would you choose deep learning over classical rule-based vision? A: For defects that cannot be specified geometrically — stains, texture variation, cosmetic scratches, class variability — where enough labelled images exist; classical methods remain better for metrology, where traceable accuracy and explainability are required.
• Q: A vision-guided robot picks accurately in the middle of the field of view but misses at the edges. What do you check? A: Lens distortion correction and the calibration coverage — calibration images probably did not include the field edges, or the distortion coefficients were not applied; also check whether the part plane is not parallel to the sensor (perspective error) and whether the pick plane height matches the calibration plane.

Common mistakes

• Treating vision as a software problem and buying the camera before deciding lighting, lens, working distance and field of view — the usual cause of a failed project.
• Confusing resolution (pixels) with accuracy — accuracy depends on contrast, optics, calibration and sub-pixel fitting, not pixel count alone.
• Measuring in pixels and forgetting the pixel-to-mm calibration, or applying a scale calibrated at one height to parts at a different height.
• Skipping lens distortion correction, then blaming the robot or the algorithm for edge-of-field errors.
• Using a rolling-shutter camera or a long exposure on moving parts and mistaking motion blur for a part defect.
• Thresholding globally when illumination is uneven, instead of using adaptive thresholding or fixing the lighting.
• Assuming camera calibration alone is enough for a robot — ignoring hand-eye calibration and the tool-centre-point definition.
• Training a CNN on a few hundred images from one shift and one lighting condition, then reporting its accuracy as the line's accuracy.

Related topics

Robot kinematics · Robot programming and ROS basics · Industrial robots and applications · Sensors and transducers in mechatronics · Digital image processing · PLC and industrial automation · Machine learning for perception