Practical Scene & Object Reconstruction

How do we push neural reconstruction toward production-quality at the scales and fidelities real graphics applications need — large scenes, unstructured camera arrays, physically-faithful shading?

Neural scene representations (NeRFs, Gaussian splats) capture stunning visual fidelity, but typically work on small bounded scenes captured under controlled conditions. Pushing the same machinery toward production graphics requires solving practical bottlenecks: scaling to large indoor and outdoor scenes, handling unstructured (non-rig) capture, integrating physically-correct light transport, and stabilising the optimisation when geometry is otherwise ambiguous.

A long collaboration with Weiwei Xu and Hujun Bao at Zhejiang has pushed neural reconstruction at one bottleneck per year — distributed tile-MLPs for large indoor scenes (SNISR, SIGGRAPH 2022), bundle-adjusting NeRFs with ADMM consensus over tiles at large scale (ScaNeRF, SIGGRAPH Asia 2023), local Gaussian density mixtures for unstructured capture with curved-surface reflections (LGDM, SIGGRAPH Asia 2024), and differentiable area-light shading for material recovery (EOR, SIGGRAPH Asia 2025). Shape from Tracing (3DV 2020) sits at the head of the line: an early step that used differentiable path tracing — full global illumination, not just shading — as the forward model for joint geometry and SVBRDF recovery.

Authors

Hujun Bao · Bach-Thuan Bui · Dongyoung Choi · Jaemin Cho · Loudon Cohen · Zheng Dong · Michael Fairley · Yaoan Gao · Purvi Goel · James Guesman · Hyunho Ha · Qixing Huang · Hyeonjoong Jang · Woohyun Kang · Hakyeong Kim · Min H. Kim · Andreas Meuleman · Minh-Hieu Nguyen · Yifan Peng · Daniel Ritchie · Belal Shaheen · Yujun Shen · Shubham · Vikas Thamizharasan · Chi Wang · Huamin Wang · Qi Wang · Michael Wu · Tim Wu · Xiuchao Wu · Jiamin Xu · Weiwei Xu · Matthew David Zane · Xin Zhang · Zihan Zhu · Changqing Zou

Papers in this thread

Shape from Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing

International Conference on 3D Vision (3DV), 2020

Uses differentiable path tracing — with global illumination effects like interreflection in the forward model — to refine a coarse mesh and its per-facet SVBRDF, so shading, shadow, and material are jointly disambiguated from images captured by phone and consumer 360 camera.

Scalable Neural Indoor Scene Rendering

ACM Transactions on Graphics (SIGGRAPH), 2022

Tiles a large indoor scene and assigns a small MLP per tile, with a separate view-dependent branch for reflections, so training distributes across GPUs and rendering stays interactive — at scenes over 100 square meters.

ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene Rendering

ACM Transactions on Graphics (SIGGRAPH Asia), 2023

Pushes the tiled-NeRF idea into the bundle-adjusting regime: each tile carries a hash grid plus diffuse and specular MLPs, and ADMM reaches camera-pose consensus across tiles, with a specular-aware warping loss giving the poses a second optimization path.

Local Gaussian Density Mixtures for Unstructured Lumigraph Rendering

SIGGRAPH Asia, 2024

Targets curved-surface reflections and refractions — exactly where view-consistent global density models break — with a per-view Gaussian-mixture density along each ray, then warps and fuses these local volumes with learned blending weights for unstructured lumigraph rendering.

Efficient Object Reconstruction with Differentiable Area Light Shading

SIGGRAPH Asia, 2025

Replaces point lights with active area lighting during capture, then differentiates through linearly transformed cosines plus shadow visibility weighting for shading — recovering material at +3 dB relighting PSNR or matching point-light quality from a fifth of the photos.

Related papers

OmniSDF: Scene Reconstruction using Omnidirectional Signed Distance Functions and Adaptive Binoctrees

Computer Vision and Pattern Recognition (CVPR), 2024

Reconstructs scenes captured by a small-baseline circular sweep of a 360 camera by placing an SDF inside an adaptively subdivided spherical binoctree, whose geometry matches the capture setting and keeps memory in line with detail.

Splat-based Gradient-domain Fusion for Seamless View Transition

3D Vision, 2026

Tackles the sparse-view, wide-baseline regime where Gaussian splatting drops geometry — anchors it with two-view stereo, fills intermediate viewpoints via reprojection, and fuses in the gradient domain so color transitions stay smooth across views.

TreeDGS: Aerial Gaussian Splatting for Distant DBH Measurement

MDPI Remote Sensing, 2026

Treats aerial Gaussian splatting as a measurement instrument: trunks span only a few pixels from altitude, so the method extracts a dense opacity-weighted point set, isolates trunk samples, and fits solid circles to estimate diameter-at-breast-height — beating a LiDAR baseline at 4.79 cm RMSE.

Monocular Dynamic 3D Reconstruction

When the input is only ordinary RGB video — no depth sensor, no rig — can we recover dynamic 3D scene geometry well enough to compete with depth-sensor-supervised methods?

Monocular dynamic 3D reconstruction takes a single moving camera observing a deforming scene and tries to recover a complete 4D representation — geometry, appearance, motion — over the captured time window. The problem is fundamentally under-constrained at any one instant, and progress depends on how well the chosen scene representation and the supervision signals work together.

Yiqing Liang's PhD has driven this arc. Starting from semantic attention flow fields built atop a dynamic NeRF at ICCV 2023, the work moved to a forward-warping Gaussian deformation formulation (GauFRe, with Meta colleagues) for real-time rendering, then to a TMLR benchmark (MonoDyGauBench) that puts the recent flood of monocular dynamic Gaussian methods on a like-for-like footing. The latest paper (Zero-MSF, with NVIDIA) abandons per-scene optimization entirely and trains a feed-forward predictor for scene flow that generalizes zero-shot to in-the-wild video.

Authors

Abhishek Badki · Orazio Gallo · Leonidas J. Guibas · Adam Harley · Numair Khan · Eliot Laidlaw · Douglas Lanman · Yiqing Liang · Runfeng Li · Zhengqin Li · Alexander Meyerowitz · Thu Nguyen-Phuoc · Mikhail Okunev · Srinath Sridhar · Hang Su · Mikaela Angelina Uy · Lei Xiao

Papers in this thread

Semantic Attention Flow Fields

International Conference on Computer Vision (ICCV), 2023

Reconstructs a 4D neural volume carrying not just color and density but also scene flow, semantics, and attention, then uses the latter two to decompose foreground objects from background across spacetime without supervision.

GauFRe🧇: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis

arXiv (Dec.~2023) + WACV, 2025

Casts monocular dynamic reconstruction as a canonical Gaussian template plus a forward-warping deformation field, with a separate static component initialized to absorb non-moving regions so the deformation focuses on what actually moves. Trains in roughly twenty minutes and renders in real time.

Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules

Transactions on Machine Learning Research, 2025

An apples-to-apples benchmark of monocular dynamic Gaussian splatting methods, categorized by motion representation. Method differences are resolvable on synthetic data but get swamped by real-world scene complexity, and the optimization is uniformly brittle.

Zero-Shot Monocular Scene Flow Estimation in the Wild

Computer Vision and Pattern Recognition (CVPR), 2025

A feed-forward model that jointly predicts geometry and scene flow, trained on a one-million-sample synthetic recipe. Generalizes zero-shot to casual DAVIS video and RoboTAP manipulation scenes — no per-scene optimization required.

Active Illumination for Dynamic 3D Reconstruction

Physical modelling of active illumination from raw sensor measurements can improve scene estimation and avoid errors from derived depth.

Time-of-flight and structured-light cameras are typically used as depth sensors: their raw measurements are processed into a per-pixel depth map, and downstream reconstruction methods treat that depth as input. But, depth processing often simplifies scene assumptions, creating noise in low-reflectance regions, flying pixels with multi-path interference, and motion artifacts in fast-moving scenes from requiring multiple illumination readings for depth estimates. Further, it is difficult to integrate these raw measurements with other sensor modalities, like colour cameras.

This line of work rethinks reconstruction for heterogeneous multi-shot imaging processes. Built upon a differentiable forward model of how the active illumination produces the raw sensor output for a given scene, these methods optimise a 4D volumetric scene representation (like NeRF or 3DGS) so that rendered measurements match what the sensor captured. This lets us principally integrate sensor measurements over spacetime, including across modalities, to reduce noise, resolve ambiguities in multi-shot sensing, and improve robustness to multi-path interference. And, as we model motion over time, then we can estimate and resample fast motion like swinging baseball bats to slow motion.

Authors

Benjamin Attal · Anh Duong · Aaron Gokaslan · Zixuan Guo · Changil Kim · Hakyeong Kim · Min H. Kim · Eliot Laidlaw · Runfeng Li · Marc Mapeke · Andreas Meuleman · Matthew O'Toole · Mikhail Okunev · Christian Richardt · Aarrushi Shandilya

Papers in this thread

TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis

Advances in Neural Information Processing Systems (NeurIPS), 2021

Established that a 4D scene can be supervised directly by continuous-wave ToF phasor measurements rather than processed depth, with added color cameras, showing low noise, superresolution, and better multi-path handling.

Flowed Time of Flight Radiance Fields

European Conference on Computer Vision (ECCV), 2024

Adds motion vectors that are jointly estimated with geometry. Uses four raw frames (not phasors) captured over time from a continuous-wave ToF sensor to create a coherent dynamic reconstruction. 20× less depth error on dynamic objects than the C-ToF baseline.

Time of the Flight of the Gaussians

Computer Vision and Pattern Recognition (CVPR), 2025

Applies raw ToF supervision to a Gaussian splatting backbone, with two heuristics that stabilise the otherwise-brittle 3DGS optimisation when depth is not directly measured. Comparable quality to neural volumetric baselines while training ~100× faster.

Related papers

Neural Fields for Structured Lighting

International Conference on Computer Vision (ICCV), 2023

Carries the supervision-by-raw-measurement approach from ToF over to structured light, and lets us separate direct and ambient illumination. Recovers higher-fidelity depth on objects than commodity structured light sensors, including for partially-transparent surfaces.

FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

European Conference on Computer Vision (ECCV), 2022

Fuses ToF depth with stereo from a smartphone's optically-stabilised main RGB camera, where the floating lens has unknown pose. Self-calibrates the multi-sensor geometry from a single snapshot, then fuses via a correlation volume.

Controllable Generative Models

How do we efficiently control generative models to produce what we want — preserving identity, 3D structure, style — without sacrificing quality?

A generative model that can sample new content is impressive; one that produces exactly what a user has in mind is useful. Controlling generation requires aligning the model's latent structure with axes a person can articulate — identity, pose, style, lighting, geometry — without sacrificing the photorealism that brought the model to relevance in the first place. There is usually a quality-versus-control tradeoff to manage.

The thread runs from Youssef Mejjati's PhD work on unsupervised attention for image-to-image translation, through compositional controls (object stamps, GaussiGAN's 3D Gaussian primitives from silhouettes alone), into 3DMM-conditioned face generation where Yiwen Huang's PhD now sits. Two recent moves matter: TaxFreeGAN closes the FID gap to unconditional StyleGAN under 3DMM conditioning, and the disentangling-3D work shows that the noise in CLIP's embedding space — not the disentanglement strategy — is what kills quality. R3GAN sits alongside this arc as the architectural reset: a principled relativistic loss that lets the modern GAN drop its bag of tricks.

Authors

Akin Caliskan · Darren Cosker · Aaron Gokaslan · Yiwen Huang · Berkay Kicanaoglu · Hyeongwoo Kim · Kwang In Kim · Atsunobu Kotani · Volodymyr Kuleshov · Youssef A. Mejjati · Isa Milefchik · Christian Richardt · Zejiang Shen · Michael Snower · Stefanie Tellex · Vikas Thamizharasan · Oliver Wang · Yue Wang · Xinjie Yi · Zhiqiu Yu · Qian Zhang

Papers in this thread

Unsupervised Attention-guided Image to Image Translation

Neural Information Processing Systems (NeurIPS), 2018

Jointly trains attention with generators and discriminators so unsupervised image-to-image translation can localize edits to objects without disturbing background or inter-object structure.

Generating Handwriting via Decoupled Style Descriptors

European Conference on Computer Vision (ECCV), 2020

Factors handwriting style into separate character-level and writer-level descriptors, letting the model generate new characters in a held-out writer's hand from only a few samples.

Generating Object Stamps

CVPR Workshop on AI for Content Creation, 2020

Splits conditional object insertion into a mask generator (shape, given a class and bounding box) and a texture generator (appearance, conditioned on the background), so the inserted object is both diverse in shape and consistent with its surroundings.

GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes

BMVC 2021 and CVPR Workshop on AI for Content Creation, 2021

Learns a coarse 3D object representation as a set of self-supervised anisotropic 3D Gaussians from unposed 2D masks alone, then uses it to drive controllable mask and texture synthesis with interactive posing.

Learning Physically-based Face Material and Lighting Decomposition

International Conference on Computational Visual Media, 2022

Estimates per-portrait surface normals, albedo, roughness, and a high-frequency lighting map, and decomposes diffuse and specular reflectance — so a downstream editor can relight a face from a single photograph.

Removing the Quality Tax in Controllable Face Generation

Winter Conference on Applications of Computer Vision (WACV) and AI for Content Creation (AI4CC) @ CVPR 2023, 2024

Formalizes 3DMM-conditioned face generation as a math problem, then applies targeted fixes that close the FID gap to unconditional StyleGAN — so controllability no longer costs visible image quality.

Disentangling 3D from Large Vision-Language Models for Controlled Portrait Generation

2024

Disentangles 3D portrait generation from a frozen CLIP plus a FLAME morphable model, then identifies CLIP's noisy embedding directions as the residual source of entanglement and damps them with a stochastic Jacobian regularizer.

The GAN is Dead; Long Live the GAN! A Modern GAN Baseline

Neural Information Processing Systems (NeurIPS), 2024

A regularized relativistic GAN loss with proven local convergence lets a minimalist StyleGAN2-derived architecture — stripped of the usual stabilization tricks — beat StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST, and compete with diffusion models.

Light Fields — from Display to 4D Algorithms

The light field is a 4D record of a scene's rays — how do we present it to humans, interact with it, and process it computationally?

A light field captures the radiance at every point in space, in every direction — a 4D function that fully describes how light fills a scene. Captured light fields enable refocusing, depth recovery, and parallax view synthesis; displayed light fields offer glasses-free 3D. The challenge is data density: 4D content stresses capture devices, display hardware, and processing pipelines.

Two sub-arcs sit in this thread. The first (2012–2015) targets light field displays — an Emerging Technologies demo of painting directly into a glasses-free 3D display, content-adaptive lenticular prints that reshape the lenslet array to the captured light field, and a UIST paper that turns that lenslet array into a joint display-and-pen-input surface. The second (2019–2021), led by Numair Khan with Min H. Kim at KAIST, develops dense algorithms over captured 4D content: view-consistent superpixels via epipolar-plane image segmentation, edge-aware bidirectional diffusion for depth, and a differentiable diffusion routine for sparse-to-dense depth from multi-view images.

Authors

Marc Alexa · Simon Heinzle · Stanislav Jakuschevskij · Lucas Kasser · Jan Kautz · Numair Khan · Min H. Kim · Wojciech Matusik · James McCann · Jim McCann · Samuel Muff · Hanspeter Pfister · Henry Stone · Qian Zhang

Papers in this thread

Interactive Light Field Painting

SIGGRAPH Emerging Technologies, 2012

An early SIGGRAPH Emerging Technologies demo of a dual-purpose lenslet array that both displays a light field and senses a 3D light-pen position — the live precursor to the UIST 2015 write-up.

Content-adaptive Lenticular Prints

ACM Transactions on Graphics (SIGGRAPH), 2013

Treats the lenslet array as something to optimise rather than fix in advance — given an input light field, solve for lenslet size, shape, and arrangement that trade spatial against angular resolution where it matters. Validated by 3D printing the resulting arrays.

Joint 5D Pen Input for Light Field Displays

User Interface Software and Technology (UIST), 2015

One lenslet array does double duty — light field output and 5D pen input (3D position plus 2D orientation) at 150 Hz, with millimetre-scale accuracy. The display surface and the input surface are the same surface.

View-consistent 4D Light Field Superpixel Segmentation

International Conference on Computer Vision (ICCV), 2019

Segments horizontal and vertical EPIs first, then clusters and propagates across all sub-aperture views — so superpixels stay consistent and respect occlusion as the viewpoint shifts, rather than being propagated outward from a single central view.

Differentiable Diffusion for Dense Depth Estimation from Multi-view Images

Computer Vision and Pattern Recognition (CVPR), 2021

Dense depth is obtained by diffusing a sparse set of points whose positions, depths, and weights are differentiably optimised through Gaussian splatting against a multi-view RGB reprojection loss. Scales to the 50k+ points needed for non-trivial scenes.

4D Light Field Depth Estimation

BMVC, 2021

A pair of BMVC papers that estimate 4D depth from sparse EPI-derived edges and diffuse them outward — the 2021 paper separates depth from texture edges via bidirectional diffusion, and the 2020 paper propagates the central-view depth to every other sub-aperture view in an occlusion-aware way.

Editing Video by Recovering Scene Structure

How do we let users edit captured video meaningfully — by first recovering the scene structure (moving objects, lighting vs. reflectance, cross-frame consistency) that makes plausible modifications possible?

Editing video is harder than editing a photograph: changes to one frame must propagate consistently to every other, and many edits (removing a person, separating lighting from material, stabilising flicker) require understanding the underlying scene rather than just manipulating pixels. The papers in this thread approach editing as inverse reconstruction: decompose video into scene structure first, then edit.

A postdoc-era thread spanning UCL, MPI-Inf, Harvard, and LIRIS-CNRS. The earliest piece (2011, UCL) is the cinemagraphs authoring tool — a moment image isolated from a stabilised clip. Miguel Granados led the video-inpainting work at MPI-Inf (2012) — removing dynamic objects from crowded scenes, and the harder case of background recovery under a free-moving camera. Nicolas Bonneel led the consistency and decomposition line (2014–2017) — interactive intrinsic decomposition, blind temporal consistency stabilising any per-frame filter, and the spatio-temporal extension to camera arrays. The 2016 multicut paper takes a different angle on the same theme: cut the video into the right regions before editing.

Authors

Bjoern Andres · Nicolas Bonneel · Miguel Granados · Oliver Grau · Jan Kautz · Kwang In Kim · Steffen Kirchhoff · Evgeny Levinkov · Sylvain Paris · Fabrizio Pece · Hanspeter Pfister · Kartic Subr · Kalyan Sunkavalli · Deqing Sun · Christian Theobalt · Oliver Wang

Papers in this thread

Towards Moment Images: Automatic Cinemagraphs

European Conference on Visual Media Production (CVMP), 2011

An authoring tool that pipelines stabilisation, segmentation, motion selection, and loop detection to produce cinemagraphs — short looping clips where only a chosen region moves.

How Not to Be Seen - Object Removal from Videos of Crowded Scenes

Computer Graphics Forum (Eurographics), 2012

Object removal from crowded scenes by filling the spatio-temporal hole from other regions of the video where the occluded background was visible, posed as a graph-cut optimisation. Pitched at occlusions harder than previous work had attempted.

Background Inpainting for Videos with Dynamic Objects and a Free-moving Camera

European Conference on Computer Vision (ECCV), 2012

Inpaints background revealed by removing dynamic objects from a free-moving-camera video by aligning candidate frames with piecewise planar homographies — sidestepping the full per-frame depth and pose recovery that earlier free-camera methods required.

Interactive Intrinsic Video Editing

ACM Transactions on Graphics (SIGGRAPH Asia), 2014

Decomposes video into reflectance and illumination via a hybrid L2-Lp gradient split, fast enough (two orders of magnitude over prior tools) to support interactive refinement and lighting-aware compositing.

Blind Video Temporal Consistency

ACM Transactions on Graphics (SIGGRAPH Asia), 2015

A gradient-domain post-process that stabilises any per-frame filter against flicker by borrowing temporal regularity from the unprocessed video — agnostic to what the filter actually is. Demonstrated across stylisation, intrinsic decomposition, and depth.

Interactive Multicut Video Segmentation

Pacific Graphics 2016 (Short Paper), 2016

Interactive multi-label video segmentation from multi-coloured scribbles, posed as a multicut on a supervoxel graph and solved fast enough to feel responsive. Multiple objects cut at once with consistent spatio-temporal boundaries, rather than chained binary segmentations.

Consistent Video Filtering for Camera Arrays

Computer Graphics Forum (Eurographics), 2017

Extends the blind-consistency idea from time to time-and-space across stereo, light field, and wide-baseline rigs, and adds a filter-transfer scheme that runs the expensive filter on a small subset of frames and propagates the effect — an order-of-magnitude saving for camera-array data.