VFX: compositing, rotoscoping and matchmoving

In short

Compositing is the craft of combining separately created images - live-action plates, CG renders, matte paintings, particles - into one seamless frame that looks as though a single camera shot it. Rotoscoping (hand-drawn, animated mattes) and matchmoving (recovering the real camera's motion and lens so CG sits in the plate) are the two support disciplines that make compositing possible when a key or an automatic solve will not do. These three skills are the entry gate to the Indian VFX industry - roto/paint and prep are the standard first jobs at DNEG India, MPC, Technicolor, Framestore Mumbai, Red Chillies VFX, Prime Focus, NY VFXWaala, Phantom FX and Makuta, and university B.Sc/B.Des papers on "Visual Effects" and "Digital Compositing" examine keying theory, alpha maths, matte generation and camera solving. Practically every Indian feature, OTT series and ad today passes through a compositing script, so a graduate is judged on edge quality, colour maths and shot-to-shot consistency more than on software menus.

Explained

1. The compositing model: alpha, over and premultiplication

• A digital image for VFX has four channels - R, G, B and A (alpha). Alpha stores *coverage*: 0 = fully transparent, 1 = fully opaque, fractional values at soft edges, hair and motion blur.
• The fundamental operation is the over (Porter-Duff) composite: the foreground is laid over the background in proportion to alpha.
• Straight (unpremultiplied) images store the true colour of the object in RGB regardless of alpha; premultiplied images store RGB already multiplied by alpha. Renders from 3D (OpenEXR) are normally premultiplied.
• Rule: colour-correct, grade or blur a foreground only after unpremultiplying (divide RGB by A), then premultiply again before the over - otherwise edges darken or develop a bright fringe.
• Compositing orders: over, in, out, atop, xor, plus/add (lights, flares, glows - no alpha needed), multiply (shadows, dirt), screen, mask/stencil. Add and over are the two most used.
• Node-based vs layer-based: Nuke, Fusion and Natron are node graphs (a DAG - directed acyclic graph) where each node is an operation; After Effects and older Combustion are layer stacks. Node graphs scale to hundreds of elements, allow re-use of a branch, and are the industry standard for film.

2. Keying, despill and prep

• Keying extracts a matte automatically from a coloured backing. Types:
  • Luma key - threshold on brightness; crude, used for bright elements on black (fire, sparks, practical lights).
  • Chroma key - selects a hue/saturation range; cheap, hard-edged, poor on hair.
  • Colour-difference key - classic green-screen maths: alpha derived from G − max(R, B) (blue screen: B − max(R, G)). Basis of Primatte/Keylight-style keyers.
  • Image-based / learned keyers - IBK (image-based keyer in Nuke) builds a synthetic clean screen from the plate itself; machine-learning keyers and roto-assist tools are now common in Indian facilities but output is always checked by an artist.
• Why green: digital sensors have twice as many green photosites in a Bayer pattern, so green carries the most luminance detail and the least noise; blue is chosen for blond hair, smoke or when wardrobe is green.
• Shooting discipline (what prep artists ask DOPs for): evenly lit backing, 2-3 stops of separation between subject and screen, subject kept several metres off the screen to limit spill, tracking markers only where needed, a clean plate (the set with no actor) and a grey/chrome ball plus colour chart (Macbeth/ColorChecker) for lighting reference.
• Despill: green bounce on skin and hair must be removed and the lost colour replaced; a despill either suppresses the green channel toward the average of R and B or re-introduces a sampled colour.
• Multi-pass keying: separate core matte (solid interior), edge matte (soft detail) and holdout/garbage matte, then combine - a single key rarely satisfies both density and edge detail.
• Prep/paint work: wire and rig removal, marker and tracker removal, dust-busting, scratch and sensor-dust clean-up, de-noise before key and re-grain after.

3. Rotoscoping

• Definition: creating an animated matte by hand, frame by frame, around an object that cannot be keyed - a shot on location, a prop crossing the screen, an actor in front of real scenery.
• Historical origin: Max Fleischer's 1915 rotoscope traced live-action frames for animation; the word survives for the matte-drawing craft.
• Tools: Silhouette, Mocha Pro, Nuke's RotoPaint, Fusion, After Effects masks. Shapes are Bézier (control points with tangent handles) or B-spline (control points off the curve, smoother, fewer points).
• Workflow:
  1. Study the whole shot; identify the keyframes where the shape changes direction or silhouette.
  2. Break the subject into separate shapes per rigid part (upper arm, forearm, hand) - never one giant shape for a body.
  3. Draw on an extreme frame, then the other extreme, then bisect (keyframe on 1s only where motion demands it).
  4. Animate shape *and* transform (translate/rotate/scale) separately - moving the whole shape with a transform keeps the point animation clean.
  5. Add per-point or per-shape motion blur and feather to match the plate; a razor-sharp roto edge on a blurred arm reads as fake instantly.
  6. Check in motion, at full resolution, over a contrasting colour, and check the matte channel alone.
• Planar tracking (Mocha) tracks a flat surface's four-corner homography and carries a spline with it, cutting hand-work drastically on walls, screens, floors, signage and billboards.
• Quality terms: *chatter* (edge jitter between frames), *crawling*, *lagging* (shape behind the subject), *clipping* (cutting into the subject), *ghosting*.
• Limitations: labour-intensive (a single complex shot can take days); fails on fine hair, smoke, water, glass - these need keys, hair tools or re-shoots.

4. Matchmoving, camera tracking and 3D solving

• Matchmove = recovering the real camera's motion, focal length and lens distortion so that CG can be rendered through a virtual camera that matches the plate exactly. Also called camera tracking; the related jobs are object tracking (a moving prop or vehicle) and body/match-animate (a rough geometry double of an actor).
• 2D tracking: follow a feature's x, y over time; a *one-point* track gives position, *two points* give position + rotation + scale, *four points* give a planar/corner-pin (perspective) warp. Used for screen inserts, stabilisation, bolting an element to a surface.
• 2.5D: a 2D track plus depth-sorted cards - set extensions and parallax-light shots.
• 3D camera solve (SynthEyes, 3DEqualizer, PFTrack, Nuke CameraTracker, Blender):
  1. Load the plate at full resolution with correct pixel aspect and frame rate; de-interlace/de-noise if needed.
  2. Enter known camera data from the camera report: sensor/filmback size, focal length, frame rate, lens.
  3. Auto-track features, then delete bad trackers (on moving subjects, specular highlights, screen edges).
  4. Solve - the software uses bundle adjustment/structure-from-motion to find camera pose per frame plus a 3D point cloud.
  5. Undistort or solve lens distortion; survey/LiDAR data and known set measurements set true scale and orientation (a solve from images alone has an unknown absolute scale).
  6. Validate by projecting the point cloud back - report reprojection error in pixels; under ~0.5 px is generally considered a good film solve, and sub-pixel is the target, though every facility sets its own tolerance.
  7. Export camera + point cloud + ground plane (FBX/Alembic) to lighting and animation.
• Perspective geometry: a pinhole camera projects a 3D point by the collinearity relations below; parallax (apparent relative motion of near and far features) is what makes a solve possible - a pure pan/tilt from a tripod has no parallax, so only a nodal/rotation-only solve is possible and no 3D depth can be recovered.
• Failure cases: motion blur, rolling shutter on CMOS cameras, zoom during the shot (focal length becomes an unknown per frame), featureless surfaces, a locked-off nodal pan, heavy depth-of-field.

5. Colour management, formats and the linear workflow

• Compositing maths (add, multiply, blur, defocus, motion blur) is physically correct only in scene-linear light. So: convert camera log footage (ARRI LogC, Sony S-Log3, RED Log3G10) to linear, composite, then view through a display transform.
• ACES (Academy Colour Encoding System) is the common standard: IDT/input transform → linear ACES working space → ODT/output transform (Rec.709 for TV, DCI-P3 for cinema). Facilities in India working for studios deliver to ACES or to a studio-supplied LUT.
• Bit depth: 8-bit integer (web/pre-vis, 256 levels per channel), 10/12-bit log (broadcast), 16-bit half float and 32-bit float (compositing; floats hold values above 1.0, needed for highlights and HDR).
• Formats: OpenEXR - the compositing standard (multi-channel, multi-part, float, lossless Zip/PIZ or lossy DWAA/DWAB compression, holds render passes and deep data); DPX (10-bit log film/DI); TIFF; PNG/JPEG only for reference. Deliverables often as DPX/EXR sequences or ProRes/DNxHR for review.
• Render passes (AOVs): diffuse, specular, reflection, refraction, subsurface, emission, shadow, ambient occlusion, plus data passes Z-depth, normals, motion vectors, position (P), cryptomatte/ID. The beauty is re-buildable by adding the light passes - this lets a compositor re-balance a CG render without re-rendering.
• Deep compositing: every pixel stores samples at multiple depths, so holdouts between CG elements and volumetrics resolve automatically without matte line errors.
• Integration toolkit: match black level and contrast, grain/noise (de-grain, composite, re-grain), lens defocus (not a Gaussian blur - bokeh shape matters), chromatic aberration, lens distortion re-apply, flares, light wrap, edge blend, interactive light, shadows and contact shadows, atmospheric haze by depth.

6. Pipeline, roles and the Indian production context

• Shot path: plate ingest → conform/colour → prep/paint → roto → matchmove → layout → CG (model/texture/rig/anim/FX/light) → compositing → review (dailies) → DI/finishing.
• Roles and typical Indian entry points: Roto Artist and Prep/Paint Artist (most common fresher role), Matchmove/Tracking Artist, Junior Compositor, then Compositor → Senior → Lead → Comp Supervisor → VFX Supervisor. Allied: Layout, FX, Lighting, Stereo conversion (India does a large share of the world's 2D-to-3D stereo work), Pipeline TD.
• Review language: *plate*, *slap comp*, *temp*, *wedge* (a set of variants), *client note*, *version up* (v001, v002), *turnover*, *delivery*.
• Pipeline software: ShotGrid/Flow, ftrack, Kitsu for shot tracking; OpenColorIO for colour; Deadline/Tractor for render farms; Perforce/Git for code and tools; USD and Alembic for scene interchange.
• Legal and standards context in India: VFX work is a *work made for hire* under contract, with copyright under the Copyright Act, 1957 (as amended in 2012) - the producer usually owns the output and the artist's rights are contractual; signing NDAs and working on an MPA/TPN (Trusted Partner Network) assessed secure network (no phones, no internet on artist machines, watermarked review) is normal at the large Mumbai, Chennai, Hyderabad and Bengaluru facilities. Content classification/certification of the finished film is under the Cinematograph Act, 1952 as amended by the Cinematograph (Amendment) Act, 2023 (U, UA 7+/13+/16+, A). Employment terms now sit under the four Labour Codes, in force from 21 November 2025. Broadcast delivery uses Rec.709 HD / Rec.2020 UHD; cinema uses DCI-P3 in a DCP. The AVGC-XR sector has a dedicated national task force and several state AVGC policies (Karnataka, Maharashtra, Telangana, Tamil Nadu, Kerala) offering incentives - the details differ by state and change, so verify the current scheme before quoting it.

Key points

• Alpha - per-pixel coverage, 0-1; the whole of compositing rests on it.
• Over operator - out = Fg + (1 − αFg)·Bg with premultiplied inputs; the default merge.
• Premultiply/unpremultiply - divide by alpha before a grade, multiply back after, or edges go dark/bright.
• Add (plus) - correct for light-emitting elements (flares, glows, fire); over is for solid objects.
• Colour-difference key - green-screen alpha from G − max(R, B); green is used because Bayer sensors have most green samples.
• Despill - removing screen bounce from the subject and restoring the correct colour; forgetting it is the most visible keying error.
• Core / edge / garbage matte - three mattes combined; one key cannot be both dense and soft-edged.
• Clean plate - the empty set, shot on the same camera and lighting; the single most useful item for paint and roto.
• Rotoscoping - hand-animated matte, split into rigid parts, with motion blur and feather matched to the plate.
• Bézier vs B-spline - handles on the curve vs control points off it; B-splines give smoother organic shapes with fewer points.
• Planar tracking - solves a 4-corner homography for a flat surface and carries splines/inserts with it.
• Matchmove - recovering camera position, rotation, focal length and distortion per frame; validated by reprojection error in pixels.
• Parallax - needed for a 3D solve; a nodal (tripod pan) shot gives rotation only and no depth.
• Scene-linear workflow - composite in linear float, view through an ACES/LUT display transform; maths is wrong in log or sRGB.
• OpenEXR 16/32-bit float - the compositing format; holds AOVs, deep data and values above 1.0.
• AOVs/render passes - diffuse, spec, reflection, AO plus data passes (Z, normals, motion vectors, cryptomatte) for comp-side control.
• Integration - grain, defocus, lens distortion, black level, light wrap, contact shadow: the realism comes from these, not from the key.
• Deep compositing - depth samples per pixel so holdouts and volumetrics resolve without matte lines.

Formulas and examples

Example 1 - over with premultiplied values. A foreground pixel has α = 0.4 and premultiplied RGB = (0.20, 0.12, 0.08) over a background (0.50, 0.60, 0.70), α_bg = 1.

• R: 0.20 + (1 − 0.4)(0.50) = 0.20 + 0.30 = 0.50
• G: 0.12 + 0.6 × 0.60 = 0.12 + 0.36 = 0.48
• B: 0.08 + 0.6 × 0.70 = 0.08 + 0.42 = 0.50
• α_out = 0.4 + 0.6 × 1 = 1.0
• Result (0.50, 0.48, 0.50), fully opaque.

Example 2 - why you must unpremultiply before a grade. Same pixel, α = 0.4, premultiplied RGB (0.20, 0.12, 0.08). The object's true colour is (0.20/0.4, 0.12/0.4, 0.08/0.4) = (0.50, 0.30, 0.20). Apply a 1.5× gain:

• Correct way: grade straight colour → (0.75, 0.45, 0.30), re-premultiply by 0.4 → (0.300, 0.180, 0.120).
• Both routes give the same answer for a pure *multiply*. Now apply a lift of +0.10 instead:
• Correct: straight (0.50, 0.30, 0.20) + 0.10 = (0.60, 0.40, 0.30), ×0.4 → (0.240, 0.160, 0.120).
• Wrong (lift applied to premultiplied data): (0.30, 0.22, 0.18) - the alpha-0 surround is also lifted to 0.10, so transparent areas glow and the edge gains a fringe. Multiplicative operations are alpha-safe; anything that adds a constant is not.

Example 3 - green-screen key by hand. Plate pixels, with k = 2.0:

The green shirt shows why wardrobe colour matters and why a holdout matte is drawn by hand over it.

Example 4 - field of view and a matchmove sanity check. An ARRI ALEXA Mini with a 4:3 sensor region, filmback width 28.25 mm, 32 mm prime lens, plate 2,880 px wide.

• FOV_h = 2·arctan(28.25 / (2 × 32)) = 2·arctan(0.4414) = 2 × 23.81° = 47.6°.
• Pixel pitch = 28.25 / 2,880 = 0.00981 mm/px.
• A 1.8 m tall actor standing 6 m from the camera: image height = f·X/Z = 32 × 1.8 / 6,000 mm = 32 × 0.0003 = 0.0096 m = 9.6 mm → 9.6 / 0.00981 ≈ 979 px tall.
• If the solve places the CG double at Z = 6.6 m instead, its height would be 32 × 1.8/6,600 = 8.73 mm ≈ 890 px - a 9% size error that is obvious in a side-by-side, which is why scale is set from survey measurements and not guessed.

Example 5 - roto budget and reprojection error. A 120-frame shot of a walking actor needs roto of the torso, both arms (2 shapes each) and head = 6 shapes. The artist keys shapes on every 4th frame.

• Keyframes per shape = 120/4 + 1 = 31 → total keyed shapes = 6 × 31 = 186 shape keyframes.
• At an average 3 minutes per shape keyframe = 558 minutes ≈ 9.3 hours, so roughly 1.2 working days (an 8-hour day) for one shot - the realistic unit of roto planning.
• The matchmove on the same shot has 420 valid tracker observations with Σ of squared residuals = 94.5 px². E_rms = √(94.5/420) = √0.225 = 0.474 px - inside the usual sub-pixel target, so the solve is accepted.

# Typical Nuke script shape for a green-screen shot (read as a graph, top to bottom)
Read plate.exr (log) -> Colorspace/IDT to scene-linear
  -> Denoise -> Keylight/IBK ------------------\
  -> Despill ---------------------------------- Merge(over) <- Read bg_cg.exr (premultiplied)
  -> RotoPaint (garbage + holdout mattes) ----/
Merge -> Grade (match blacks/whites) -> Defocus -> LightWrap
      -> ReGrain -> LensDistortion(re-apply) -> Write dpx/exr (ODT Rec.709 for review)

Where it is used

• Feature films and OTT series: DNEG India, MPC, Framestore, Technicolor/The Mill, Red Chillies VFX, Prime Focus, Phantom FX, NY VFXWaala, Makuta and Annapurna Studios deliver roto, prep, matchmove, comp and stereo conversion for Indian and Hollywood productions out of Mumbai, Chennai, Hyderabad and Bengaluru.
• Advertising and music videos: product shots composited onto CG backgrounds, screen replacements on phones and TVs, clean-up of brand marks - fast turnarounds, usually After Effects or Nuke in smaller studios.
• Broadcast and news: virtual sets, AR graphics and chroma-key studios at national news channels and sports broadcasters; real-time keyers instead of offline comps.
• Games, virtual production and LED volumes: camera tracking technology shared with matchmoving drives in-camera VFX on LED stages (Unreal-based), now in use at a few Indian studios and ad shoots.
• Non-entertainment uses: architectural and automotive visualisation, e-commerce product compositing, medical/scientific image overlays, forensic video clean-up, and photogrammetry/structure-from-motion in surveying and drone mapping - the same solver mathematics.
• Job roles to apply for as a fresher: Roto Artist, Prep/Paint Artist, Matchmove Artist, Junior Compositor, Stereo Artist, Pipeline/VFX TD - a reel of 4-6 clean shots with before/after breakdowns matters more than marks.

Interview questions

• Q: What is the difference between premultiplied and straight alpha? A: In a premultiplied image the RGB has already been multiplied by alpha; in a straight image RGB holds the object's true colour independent of alpha. You unpremultiply before colour-correcting and premultiply again before merging.
• Q: Write the over operator. A: With premultiplied inputs, C_out = C_fg + (1 − α_fg)·C_bg and α_out = α_fg + (1 − α_fg)·α_bg.
• Q: When would you use add instead of over? A: For light-emitting elements - flares, glows, fire, sparks, atmospherics - because light adds to the scene rather than occluding it; add also needs no alpha.
• Q: Why are green screens more common than blue? A: Bayer sensors sample green twice as densely, so a green backing gives the cleanest, least noisy matte; blue is used for blond hair, green wardrobe, smoke or night exteriors where green spill would be wrong.
• Q: What is despill and why does it matter? A: Removing the green/blue bounce the screen casts on the subject and restoring the correct colour; without it the element reads as cut-out no matter how good the matte is.
• Q: Why do you need core, edge and garbage mattes? A: A single key setting cannot be both fully dense inside the subject and soft at hair and motion blur, and lights or stands in frame must simply be excluded - so three mattes are built and combined.
• Q: What is a clean plate and how is it used? A: The same framing shot with the subject or rig removed; it supplies pixels for paint-out, marker removal, IBK screen generation and set repair.
• Q: Why can't you get a 3D camera solve from a tripod pan? A: There is no parallax - the camera only rotates about its nodal point, so depth is unobservable; you can solve rotation and focal length only.
• Q: What does reprojection error tell you? A: The RMS pixel distance between tracked features and the solved 3D points projected back into the frame; low sub-pixel values indicate a trustworthy solve, though they must still be checked visually against the plate.
• Q: How do you set the correct scale of a camera solve? A: From real measurements - set survey, LiDAR, a known object size or a measured camera move; images alone leave absolute scale undetermined.
• Q: Why composite in linear float rather than 8-bit sRGB? A: Blur, defocus, motion blur and add behave like real light only when values are proportional to scene radiance, and float holds highlight values above 1.0 that 8-bit clips.
• Q: What are AOVs and why do compositors want them? A: Arbitrary output variables - diffuse, specular, reflection, AO, Z-depth, normals, motion vectors, cryptomatte - that let a compositor re-balance and isolate parts of a CG render without a re-render.
• Q: What causes a visible matte line and how do you fix it? A: Edge alpha too hard or density mismatched, or premultiply handled wrongly; fix by eroding/blurring the edge matte, adding light wrap, matching grain and re-checking the premult order.

Common mistakes

• Grading or blurring a premultiplied element without unpremultiplying - dark edges or a bright fringe appear, and examiners look for exactly this in alpha-maths questions.
• Pulling one "perfect" key and skipping despill, edge matte and holdouts, then blaming the keyer.
• Rotoscoping a figure as one huge spline instead of separate shapes per rigid part, and keying on 1s everywhere instead of at motion extremes.
• Leaving roto edges razor-sharp on motion-blurred frames, and never checking the shape in motion or the matte channel alone.
• Leaving trackers on a moving subject, a reflection or the screen edge in a camera solve, then accepting a solve because the error number looks small.
• Forgetting to undistort before 3D work and to re-apply lens distortion and grain at the end, so CG sits too clean and too straight in the plate.
• Compositing in sRGB/8-bit, ignoring the input and output transforms, and delivering in the wrong colour space or without the required handles and naming convention.

Related topics

3D animation and rigging · Lighting, shading and rendering · Colour grading and digital intermediate · Game engines: Unity and Unreal · Virtual production and LED volumes · Motion graphics and title design