Active Illumination for Dynamic 3D Reconstruction

Physical modelling of active illumination from raw sensor measurements can improve scene estimation and avoid errors from derived depth.

Time-of-flight and structured-light cameras are typically used as depth sensors: their raw measurements are processed into a per-pixel depth map, and downstream reconstruction methods treat that depth as input. But, depth processing often simplifies scene assumptions, creating noise in low-reflectance regions, flying pixels with multi-path interference, and motion artifacts in fast-moving scenes from requiring multiple illumination readings for depth estimates. Further, it is difficult to integrate these raw measurements with other sensor modalities, like colour cameras.

This line of work rethinks reconstruction for heterogeneous multi-shot imaging processes. Built upon a differentiable forward model of how the active illumination produces the raw sensor output for a given scene, these methods optimise a 4D volumetric scene representation (like NeRF or 3DGS) so that rendered measurements match what the sensor captured. This lets us principally integrate sensor measurements over spacetime, including across modalities, to reduce noise, resolve ambiguities in multi-shot sensing, and improve robustness to multi-path interference. And, as we model motion over time, then we can estimate and resample fast motion like swinging baseball bats to slow motion.

Authors

Benjamin Attal · Eliot Laidlaw · Aaron Gokaslan · Changil Kim · Christian Richardt · Matthew O'Toole · Mikhail Okunev · Marc Mapeke · Runfeng Li · Zixuan Guo · Anh Duong · Aarrushi Shandilya · Andreas Meuleman · Hakyeong Kim · Min H. Kim

Papers in this thread

TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis

Advances in Neural Information Processing Systems (NeurIPS), 2021

Established that a 4D scene can be supervised directly by continuous-wave ToF phasor measurements rather than processed depth, with added color cameras, showing low noise, superresolution, and better multi-path handling.

Flowed Time of Flight Radiance Fields

European Conference on Computer Vision (ECCV), 2024

Adds motion vectors that are jointly estimated with geometry. Uses four raw frames (not phasors) captured over time from a continuous-wave ToF sensor to create a coherent dynamic reconstruction. 20× less depth error on dynamic objects than the C-ToF baseline.

Time of the Flight of the Gaussians

Computer Vision and Pattern Recognition (CVPR), 2025

Applies raw ToF supervision to a Gaussian splatting backbone, with two heuristics that stabilise the otherwise-brittle 3DGS optimisation when depth is not directly measured. Comparable quality to neural volumetric baselines while training ~100× faster.

Related papers

Neural Fields for Structured Lighting

International Conference on Computer Vision (ICCV), 2023

Carries the supervision-by-raw-measurement approach from ToF over to structured light, and lets us separate direct and ambient illumination. Recovers higher-fidelity depth on objects than commodity structured light sensors, including for partially-transparent surfaces.

FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

European Conference on Computer Vision (ECCV), 2022

Fuses ToF depth with stereo from a smartphone's optically-stabilised main RGB camera, where the floating lens has unknown pose. Self-calibrates the multi-sensor geometry from a single snapshot, then fuses via a correlation volume.