Practical Scene & Object Reconstruction

How do we push neural reconstruction toward production-quality at the scales and fidelities real graphics applications need — large scenes, unstructured camera arrays, physically-faithful shading?

Neural scene representations (NeRFs, Gaussian splats) capture stunning visual fidelity, but typically work on small bounded scenes captured under controlled conditions. Pushing the same machinery toward production graphics requires solving practical bottlenecks: scaling to large indoor and outdoor scenes, handling unstructured (non-rig) capture, integrating physically-correct light transport, and stabilising the optimisation when geometry is otherwise ambiguous.

A long collaboration with Weiwei Xu and Hujun Bao at Zhejiang has pushed neural reconstruction at one bottleneck per year — distributed tile-MLPs for large indoor scenes (SNISR, SIGGRAPH 2022), bundle-adjusting NeRFs with ADMM consensus over tiles at large scale (ScaNeRF, SIGGRAPH Asia 2023), local Gaussian density mixtures for unstructured capture with curved-surface reflections (LGDM, SIGGRAPH Asia 2024), and differentiable area-light shading for material recovery (EOR, SIGGRAPH Asia 2025). Shape from Tracing (3DV 2020) sits at the head of the line: an early step that used differentiable path tracing — full global illumination, not just shading — as the forward model for joint geometry and SVBRDF recovery.

Authors

Hujun Bao · Bach-Thuan Bui · Dongyoung Choi · Jaemin Cho · Loudon Cohen · Zheng Dong · Michael Fairley · Yaoan Gao · Purvi Goel · James Guesman · Hyunho Ha · Qixing Huang · Hyeonjoong Jang · Woohyun Kang · Hakyeong Kim · Min H. Kim · Andreas Meuleman · Minh-Hieu Nguyen · Yifan Peng · Daniel Ritchie · Belal Shaheen · Yujun Shen · Shubham · Vikas Thamizharasan · Chi Wang · Huamin Wang · Qi Wang · Michael Wu · Tim Wu · Xiuchao Wu · Jiamin Xu · Weiwei Xu · Matthew David Zane · Xin Zhang · Zihan Zhu · Changqing Zou

Papers in this thread

Shape from Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing

International Conference on 3D Vision (3DV), 2020

Uses differentiable path tracing — with global illumination effects like interreflection in the forward model — to refine a coarse mesh and its per-facet SVBRDF, so shading, shadow, and material are jointly disambiguated from images captured by phone and consumer 360 camera.

Scalable Neural Indoor Scene Rendering

ACM Transactions on Graphics (SIGGRAPH), 2022

Tiles a large indoor scene and assigns a small MLP per tile, with a separate view-dependent branch for reflections, so training distributes across GPUs and rendering stays interactive — at scenes over 100 square meters.

ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene Rendering

ACM Transactions on Graphics (SIGGRAPH Asia), 2023

Pushes the tiled-NeRF idea into the bundle-adjusting regime: each tile carries a hash grid plus diffuse and specular MLPs, and ADMM reaches camera-pose consensus across tiles, with a specular-aware warping loss giving the poses a second optimization path.

Local Gaussian Density Mixtures for Unstructured Lumigraph Rendering

SIGGRAPH Asia, 2024

Targets curved-surface reflections and refractions — exactly where view-consistent global density models break — with a per-view Gaussian-mixture density along each ray, then warps and fuses these local volumes with learned blending weights for unstructured lumigraph rendering.

Efficient Object Reconstruction with Differentiable Area Light Shading

SIGGRAPH Asia, 2025

Replaces point lights with active area lighting during capture, then differentiates through linearly transformed cosines plus shadow visibility weighting for shading — recovering material at +3 dB relighting PSNR or matching point-light quality from a fifth of the photos.

Related papers

OmniSDF: Scene Reconstruction using Omnidirectional Signed Distance Functions and Adaptive Binoctrees

Computer Vision and Pattern Recognition (CVPR), 2024

Reconstructs scenes captured by a small-baseline circular sweep of a 360 camera by placing an SDF inside an adaptively subdivided spherical binoctree, whose geometry matches the capture setting and keeps memory in line with detail.

Splat-based Gradient-domain Fusion for Seamless View Transition

3D Vision, 2026

Tackles the sparse-view, wide-baseline regime where Gaussian splatting drops geometry — anchors it with two-view stereo, fills intermediate viewpoints via reprojection, and fuses in the gradient domain so color transitions stay smooth across views.

TreeDGS: Aerial Gaussian Splatting for Distant DBH Measurement

MDPI Remote Sensing, 2026

Treats aerial Gaussian splatting as a measurement instrument: trunks span only a few pixels from altitude, so the method extracts a dense opacity-weighted point set, isolates trunk samples, and fits solid circles to estimate diameter-at-breast-height — beating a LiDAR baseline at 4.79 cm RMSE.