Lume lighting#

Lume is Blackbody's path-traced lighting engine. It lights the set drawn in CG (the floor, the objects, broken pieces, sand, snow and mud, ropes), the smoke and steam, the cloth and grass, water and clouds by tracing light as it really travels, instead of adding up a few direct lights. Turn it on in Properties › Lume › Lighting engine: Lume (path traced). It works in every kind of scene, with or without footage; the flames are drawn as before, in its light.

Shed on fire at 4 s: the classic engine (left) and Lume (right), the fire's light bouncing round inside the shed and its smoke lit as it is.

What it adds#

• Light that bounces. Light reaches a surface after bouncing off others: a red brick wall tints the floor in front of it, the shaded side of a crate is lit by the sunny ground beside it, an object in a room is lit by the walls round it.
• Shadows as soft as each light is big. The key light's shadow sharpens where an object touches the ground and softens as it falls away, as the sun's disc makes it. Every fire light, glowing patch of hot metal and lamp casts its own shadow, through the objects and through the smoke.
• The HDRI's own sun. With an Environment (HDRI), light is aimed at its bright parts, so a small sun in the HDRI lights the set and casts sharp shadows at once (the classic engine only takes the HDRI's average colour).
• Glass, ice and jelly that bend light. A ray goes in, bounces inside while it cannot get out, and leaves through the far side, tinted by how far it went through. Coloured glass casts its shadow in its colour. Clear things refract what is behind them and reflect the sky by the true Fresnel term.
• Caustics. The light a glass ball, an ice cube or a jelly focuses, and the light a mirror (bare polished metal: a chrome ball, a steel plate) throws, is traced from the lights themselves (from all over a lamp's bulb or panel, so a big lamp's caustic is as soft as it should be; and the fire's own flickering light, focused soft and warm, as a flame is a big light), through the glass or off the metal, to where it lands and on to the camera: the bright spot in a glass ball's shadow, the patch of lamplight a chrome ball throws onto the floor, and the light those throw on round them. Where the camera sees that spot through the glass that made it, or in a mirror, it is there too, focused: the light paths also leave their light in a cache of where it lands, which the camera's paths read. A pane, a box or a shard (flat faces) casts its exact straight-through shadow instead. The fire's light is traced this way while Composite › Shadows from the fire is on; the glow of hot matter is not (its shadow through curved glass is straight through). Not yet over footage: there no light is traced through the glass, so a ball's shadow is straight through, as a pane's, without the light it focuses.
• Even grain. Each pixel's light paths are spread evenly over the pixel, the lights and the directions a bounce can take (a low-discrepancy sequence: Owen-scrambled Sobol), not at random, so the grain falls away much faster as samples are added: on the benchmark's room lit straight by a lamp, after 1024 samples, a five-hundredth of the mean squared error of random paths.
• Physically based highlights. A GGX microfacet highlight with height-correlated shadowing and Fresnel at the facet's angle. Rough floors no longer flare toward the horizon.
• The fire lighting the set. Each of the fire's point lights is picked by how much light it brings to a surface, and its shadow is traced through the objects and the smoke.
• Smoke and steam lit by everything round them. The light inside smoke is traced as it is on the set: the flames' own light, dimmed by the smoke on its way and stopped by walls, the sky, the sunlit ground and objects under it, and the light the smoke itself scatters on, bounce after bounce. Smoke glows from inside where the fire lights it and goes dark where it hides the fire; a plume over sunny grass takes its colour from underneath; thick white steam is bright all through, as many bounces carry the light deep in. The key light's shadows through smoke are as deep as the smoke makes them.
• Water traced as it is. In a liquid scene with no footage, fabric or grass, Lume traces the liquid itself: its surface reflects and refracts by the exact Fresnel terms (rough by its roughness, rippled as it moves), the set is seen through it and mirrored in it where it really is (objects standing in it no longer show a ghost of themselves), the water absorbs in its own colour over the way through and its murk scatters the light in it, foam lies on it as a white lace and spray and bubbles scatter light like the mist they are, all lit by the same light as the set. Under it, the floor takes the caustics its waves focus (traced through its real surface) and the shadow it casts, in the water's colour; from under the surface the sky shows through Snell's window. Clear, murky and dyed water (ink, milk, blood and mud clouding into it: the dye absorbs along the way in its colour and scatters where it is thick), honey and oil; not yet water over footage or beside fabric or grass, the sea's waves, lava or ice (those are drawn by the liquid's own renderer over the set Lume lit, as before).
• The footage's own light. With Lighting › Environment from the footage (and no HDRI file), Lume lights the CG set by the shot itself: what the camera sees, as a panorama round the set (past the frame's edges, the footage's light at each height carried on: its sky above and behind, its ground below), so a CG object takes the colours of the shot it is put in, from each side as the footage has them, and reflects them.
• Dispersion. With Dispersion on, glass, ice, jelly and water bend each wavelength by its own index of refraction (Cauchy's formula from the material's index and the Abbe number of glass like it: water and ice 55, glass 59, flint glass and gems 36): the rainbow fringe round a caustic, the colours a prism or a cut crystal throws, the fire of a gem. A path takes one wavelength at its first refraction, picked by how much each adds to the picture, and keeps it; over the passes the colours add up to white again exactly, so turning it on makes nothing brighter or darker.
• Panels and real lights' profiles. An area light is traced as the panel it is (a window, a softbox), its light picked over the solid angle the panel covers from each point, so its soft shadows have little grain even close up; a light with an IES profile throws its real pattern. Lights can be set in lumens.
• Cloth and grass in the same light. Fabric and grass take that light too, on the side seen and through from behind: the fire's light shadowed by the smoke and the set, the sky dimmed under a plume, light bounced up from the ground. A curtain stops the light, so the fire behind it does not light its front. (That is in a scene with a fire; in a liquid scene the cloth and the grass are lit as with the classic engine: by the key light, the sky and the lamps.)

Settings#

Light passes#

With Light passes on, a final render also writes the set's light split by where it comes from into the composite EXR, as the layers light_key (the key light), light_sky (the sky or HDRI, the physical sky too), light_fire (the fire, hot matter and anything glowing) and light_lamps (the lights in the set). Between them they hold all the light on the set Lume traced, each light's bounces and caustics with it, denoised as the picture is; the sky and the footage seen straight past the set, and the haze, are in none of them. To turn a light up or down in the comp, or recolour it to match the footage, add its layer times (the gain − 1) to the composite: twice as much key light is the composite plus light_key. (They are the set's light before the smoke and the flames in front of it. Where light came through curved glass or off a mirror from more than one kind of light at once, its share in each pass is estimated.)

In the viewer#

While you drag or play, Lume shows one light path per pixel (denoised). Once you stop, it keeps adding paths, two at a time, until Samples (viewer) is reached. The stats overlay shows how far it has got (Lume · 24/64 light paths per pixel · sharpening). Any edit starts it afresh.

A physical sky#

Lighting › Sky: Physical (sun and air) works the sky and the sun's light out from the air itself, for wherever the key light is: a deep blue sky over a high sun, paling to a haze at the horizon and darkest opposite the sun; the sun yellowing and reddening as it sets, the sky round it glowing and the light on the set fading and warming with it, all in step. The key light becomes the sun (its colour comes from the air; Key intensity is its strength above the air) and the sky lights everything and is seen behind the set. It works with both engines: the classic engine takes the sky's light as its ambient, Lume traces it as an HDRI (aiming paths at its bright glow round the sun), and the liquids reflect and refract it.

The air is modelled as Hillaire's (2020) is, as production renderers have it: its molecules scatter blue far more than red, its haze scatters forward and absorbs a little, its ozone absorbs, each thinning with height, over a round earth; the light scattered once is followed exactly, the light scattered more than once by Hillaire's sum over all orders. Against the CIE's standard clear sky (the shape measured skies are fitted to), its brightness over the sky follows the CIE's with correlations of 0.92 to 0.98; its zenith is about 4,300 cd/m² with the sun 60° up (clear skies measure 3,000 to 6,000), and the sky gives about a sixth of the light on the ground at noon, rising to most of it as the sun reaches the horizon, as measured (tests/test_sky.py).

Clouds#

In a sky scene (the cloud simulation), Lume traces the light the cloud scatters more than once: from each cell of a grid half as fine as the cloud's, rays in every direction, each followed as the cloud really scatters light (its density as drawn, its droplets' and crystals' phase, a strong forward lobe and a little back) through up to 512 scatterings, taking the sun's light where it scatters, the sky where it leaves and the ground (sunlit through the cloud's shadow) where it goes down; past that, the light the last pass left there. The grid keeps the light's mean and the way it flows, and the march scatters both toward the eye by the cloud's phase, with the sun's light scattered once added exactly. Forty passes of that converge from nothing in about a second for a final frame; the viewer carries it on from frame to frame.

Measured against Mitsuba 3 path tracing the same cumulus (every bounce, the same density, phase, sun and sky): seen from its sunlit side or from the ground beneath it, Lume's cloud is within 3% of it in all, with no bias pixel by pixel (two thirds of its pixels within 10%: much of the rest is the reference's own grain). The classic estimate (octaves of weaker extinction, a sky term from five directions) is 41% and 35% too dark there (tools/lume_bench, Clouds). Tracing the light with a cheaper diffusion shortcut instead (scattering evenly at the strength the similarity relation gives) came out 25 to 50% too bright on the sunlit side, which is why it follows the real phase.

Smoke looks as its colour says#

Classic lighting brightens smoke well beyond what the light round it could make it (its estimate of the light scattered more than once is two to five times too much inside thick smoke), so presets made with it give smoke a dark Smoke colour and it still looks grey. Lume lights smoke as it is lit, so the same smoke can look darker and sootier: set Smoke colour to what the smoke is, pale for wood, grass and paper smoke (0.6–0.9), near black for oil, rubber and plastic (0.05–0.2). Steam is near white (0.92–0.95) either way.

Accuracy#

Lume is measured against ground truth: the same scenes rendered by Mitsuba 3, a research path tracer, with Lume's own material (tools/lume_bench). On every scene it is within 0.1% of the truth in mean brightness (0.01% to 0.09%), glass, metal and their caustics included. Caustics seen through their own glass or in a mirror are measured too (a glass ball seen from above, the floor below it through the ball). A lamp's light thrown by a mirror, by glossy metal and by the glossy coat of a plastic or painted surface is traced from the lamp too, so it lands evenly instead of sparkling. The fire's own light is traced from its lights the same way, through glass and off mirrors, while Shadows from the fire is on (the benchmark has no fire, so that is not measured against Mitsuba). What it does not yet trace from the lights: the glow of hot matter (its shadow through curved glass is straight through, and its light off a mirror is found by bouncing), and any light over footage (see Caustics above). See tools/lume_bench/README.md for the scenes, the measures and the latest results.

The light in smoke is checked the same way: a lumpy blob of smoke under a low sun and a blue sky, the mean of the light reaching points inside it from every way, against Mitsuba (six 90° cameras at each point, 1024 samples a pixel). Inside thick pale smoke and steam (albedo 0.93, up to 10 per metre) with forward scattering from 0 to 0.6, and in darker smoke (albedo 0.5), Lume is within a tenth on average (a fifth at the worst point: deep inside, where little of the sun gets), and in clear air beside it within 3%; the classic engine's estimate there is 1.4 to 5 times too bright. Against exact answers (the light inside a glowing ball of flame, thin or optically thick, by Kirchhoff's law; over open ground under a sun and a sky; inside white smoke under an even sky, which is the sky's own light all through) it is within 3% (tests/test_lume_volume.py).

An area light is checked against the light a panel sends by its solid angle: a square metre panel a metre over a matte floor lights the floor under its middle within 1.5% of Lume's material integrated over the panel by hand, three quarters of what a bulb as bright straight on would (tests/test_lights.py); and against Mitsuba's emitting rectangle in the benchmark (panel: a softbox over a block and a glossy ball), where it is within 0.04% in brightness and has a third of Mitsuba's error after the same time, against its best sampler.

Water is checked against Lume's glass, which the benchmark checks against Mitsuba: a ball of clear water held in the air, traced as water (its surface out of the liquid's surface grid, the path in and out of it as a medium), is a glass ball of water's index of refraction, traced as glass (an exact sphere). Lit by the sky, the two pictures agree within 1% on the ball (0.7% pixel by pixel) and 0.2% overall (tests/test_lume_water.py).

Speed#

Plain spheres, boxes and cylinders, the floor and broken pieces are hit exactly by each ray; meshes, hollow things and sand, snow and mud are found by marching toward them. On an RTX 3090, a simple set at 1920x1080 takes a few tenths of a second for 64 paths per pixel with 4 bounces. Against Mitsuba, which traces with the GPU's ray-tracing cores, Lume is closer to the truth after the same time on nine of the ten test scenes, against whichever of Mitsuba's samplers does best on each: 0.3 to 0.9 times its error on the rooms and the glass under the sky, a ninth to a twelfth on the glossy rooms, a thirtieth with a glass ball under a lamp; on the simplest (a white ball under an even sky, a few milliseconds a frame) Mitsuba is a little ahead (1.2x). Against Mitsuba's plain random samples, from about half its error (a room full of bounced light) to a thousandth (a room lit straight by a lamp). Blackbody compiles Lume's kernel for what the set has (without the code for broken pieces, ropes, meshes or sand where there are none, a set of plain shapes runs a fifth faster), once for each new kind of set, in a few seconds. A frame with fire light and smoke shadows takes a few times longer. Final renders trace their samples in passes (four paths per pixel at 1080p, up to sixteen on smaller pictures, each pass no longer than one at 1080p), submitted a few at a time, so long renders do not stall the GPU.

Lighting the smoke adds little: its light is traced on a grid of at most 56 cells a side (light in smoke is smooth), 24 rays a cell and three bounces a frame in a final render (24 bounces for a first frame, after a cut or a jump: steam needs them), one bounce of 10 rays a frame in the viewer, settling as the frame holds still. At 960x540 on an RTX 3090: the Crates into a fire preset (a 120x144x104 grid) about 0.13 s more a final frame and 0.02 s in the viewer; Campfire 0.05 s and 0.01 s. Frame to frame, the light it finds varies by 0.1–0.2% on average (1.5% at the 99th percentile), against 512 rays a cell.

How it works#

Every pass traces a path per pixel from the camera (four in a final render). At each surface, Lume:

1. takes the light coming straight from one of the lights (next-event estimation), picked by how much light each brings there (and a share evenly): a direction in the sun's disc; a fire light or a hot-matter light, itself picked by how much light it brings; a direction within the solid angle a lamp covers; or a direction in the HDRI picked by its brightness; weighed against finding it by bouncing (multiple importance sampling, power heuristic);
2. bounces on, by the material (a GGX highlight sampled by its visible normals, or diffuse by the cosine), or through glass by its Fresnel term;
3. ends the path by Russian roulette (from the fourth bounce) once it carries little light.

The random choices are made with a low-discrepancy sequence: a pixel's samples are points of an Owen-scrambled Sobol sequence (Burley's hash-based scrambling), each choice (the place in the pixel, which light, its direction, the bounce) with its own dimension, scrambling and shuffle, kept for the camera and the first four surfaces; one number both picks a light and, rescaled, samples it, so each light's samples keep their spread.

Each pass also traces light paths from the lights for caustics (one per sixteen camera paths): from a lamp, the key light, the fire's lights (with Shadows from the fire on: one picked by its power, through the smoke on the way) or an HDRI with a sun in it (an even sky's light is found well by bouncing), aimed at the curved clear things and the mirrors, through the glass by its Fresnel terms and off the metal by its highlight, and where one lands, its light is sent along a straight line to the camera and added to the pixel it is seen in (light tracing), and followed two bounces on. Where a light path first lands, it also leaves its light in the caustic cache: cells on the surfaces (a few millimetres across, shrinking as passes add up, so their blur goes away), hashed into a table each pass starts afresh. The camera's paths leave exactly that light to these (the lights found by way of the glass and the mirrors), so nothing is counted twice: at surfaces the camera sees straight, the light paths send it to the camera; at surfaces seen through glass or in a mirror, or past the light paths' bounces, it comes from the cache. A lamp's light paths leave from points all over its bulb, not its middle. Over footage no light paths are traced yet.

The passes are added up, with each pixel's first surface (albedo, normal, distance) and how noisy it still is. The denoiser divides out the albedo so textures stay sharp, filters the light with an edge-avoiding à-trous wavelet filter (five passes, guided by facing, distance and noise), and multiplies the albedo back.

For the smoke, each frame Lume traces the light at each cell of a grid over the fire's box (lume_volume.wgsl): rays from the cell's centre, half spread evenly over every way, half aimed at the fire (each into a cone round one of the fire's blocks, picked by how much light it sends there), weighed together by the balance heuristic, so smoke far from a small fire finds it without grain. Along each ray it adds up the flames' light, dimmed by the smoke between (an optically thick flame glows at its blackbody radiance), the key light scattered toward the cell by the smoke's phase function, and the light the last pass found there, scattered on (each pass one bounce more, carried from frame to frame); where the ray ends, the sky, or a surface (the ground, an object of the set) lit by the key light (shadowed by the smoke and the objects) and by the light in front of it. Each cell keeps the mean of what it found and the way most of it comes from (an L1 spherical harmonic), so a surface facing any way gets its own share: the ray march scatters the mean toward the camera, and the cloth and the grass (lume_light.wgsl) take the light falling on each of their sides.

For water, the liquid's surface (its distance field on the surface grid, cut flat at the walls of its box and blended into open water past the open sides, as the liquid's own renderer has it) is one more thing a path can meet (lume_water.wgsl). Where a path meets it, it is reflected or refracted at a microfacet of the water's roughness, picked by the exact Fresnel term; a path in the water is absorbed along every stretch (the water's colour over its clarity), and may scatter in it first: in the murk (in open water at least what clear water scatters, 0.03 per metre), in the dye, in the bubbles, or in the air in the spray, each picked by its optical depth along the way (the whitewater field marched a cell at a time). A point it scatters at is lit and bounced on as a surface is, with the medium's phase function (the same one light pick and shadow ray in the kernel). The lights reach what is under the water through its surfaces straight, by their Fresnel terms and the absorption on the way, in the water's colour; the key light on the floor under it by the caustic map the liquid's renderer traces through the real surface each frame (its focus and its shadow, noise-free). With a background colour behind the set (no floor and sky), the floor under the water is drawn in that colour and seen only through the water and where the liquid left it wet (its film's sheen), as the liquid's own renderer draws it; the liquid's stand-ins keep their stand-in colours.

The engine is in blackbody/engine/lume.py, blackbody/engine/lume_volume.py, blackbody/engine/wgsl/lume.wgsl, blackbody/engine/wgsl/lume_volume.wgsl, blackbody/engine/wgsl/lume_water.wgsl and blackbody/engine/wgsl/lume_denoise.wgsl.

Coming next#

Lume grows in stages. Next: the sea's waves, lava and ice traced by Lume too, water over footage and beside fabric or grass, fire and water in one scene, and caustics over footage; a full spectral mode (every colour traced as wavelengths, the fire's and hot metal's light as a blackbody emits it: today their light is traced in the blackbody's colour, and wavelengths only through clear things, with Dispersion).

Liquids Lume does not trace yet (water over footage, the sea's waves, lava, ice, fire and water in one scene, and liquid scenes with fabric or grass, which the liquid's renderer sets in the liquid) are drawn by the liquid's own renderer over the set Lume lit (and the footage), as with the classic engine.

Something wrong or missing on this page? Edit it on GitHub.