Project Case Study

Deep SfM for Radiance Fields

Independent ResearchReleased 2024

Project Overview

I compared VGGSfM and MASt3R as front ends for 2D Gaussian Splatting to test whether visually dense reconstruction also provides camera poses accurate enough for Radiance Fields.

Both outputs were converted into one COLMAP-compatible pipeline and evaluated with the same renderer. The key result was that reconstruction completeness and inverse-rendering accuracy are not the same.

Core contributions

  • Built reproducible VGGSfM-to-2DGS and MASt3R-to-2DGS evaluation paths.
  • Implemented MASt3R-to-COLMAP conversion and a Viser reconstruction viewer.
  • Separated pose quality, point density, sparse-view robustness, and memory behavior.

VGGSfM vs. MASt3R

I evaluated VGGSfM and MASt3R on 5-, 10-, and 27-view image sets. Both outputs were converted to COLMAP format, inspected in one Viser tool, and used to initialize the same 2DGS path.

VGGSfM: pose-first

Joint recovery and differentiable bundle adjustment produced sparser points but more consistent poses.

MASt3R: density-first

Dense matching recovered more structure, but the camera estimates required further refinement.

Sparse-view reconstruction

Both learned pipelines reconstructed sparse scenes where the tested COLMAP configuration failed to initialize.

Pose refinement

A follow-up experiment refined MASt3R cameras during Radiance Field training and recovered a cleaner result.

Results

VGGSfM stayed within 0.01 angular distance of the COLMAP reference, while MASt3R exceeded 0.1, explaining why the denser point cloud did not always render better.

Reconstructed point clouds

MASt3R VGGSfM
Sparse-view MASt3R point cloud reconstruction Sparse-view VGGSfM point cloud reconstruction

Downstream 2DGS reconstruction

MASt3R VGGSfM

MASt3R with camera-pose refinement

The result confirmed that learned reconstruction and differentiable pose refinement can be complementary. The repository has since attracted more than 230 GitHub stars.

Previous Project 2D GS Viewer Next Project Neural Rendering in Game Engine