gsplat icon indicating copy to clipboard operation
gsplat copied to clipboard

Add prune_scale3d argument to MCMCStrategy

Open bchretien opened this issue 10 months ago • 7 comments

MCMC-based models often suffer from very large splats that reduce the overall quality, as partially discussed in #252 and https://github.com/nerfstudio-project/nerfstudio/pull/3548. This PR adds the same logic as in the default strategy, that is pruning (or in this case relocating) splats that are too large. I used the same naming convention and default values as for the default strategy for consistency.

bchretien avatar Feb 24 '25 10:02 bchretien

If it makes things clearer, I can rename prune_scale3d to relocate_scale3d.

bchretien avatar Feb 24 '25 15:02 bchretien

Have you ran this and seen if it fixes the issues mentioned on those datasets?

N-Demir avatar Mar 06 '25 19:03 N-Demir

@N-Demir: I only ran it on my datasets (that I can't share) and it improves the situation. Some users on the Discord server tested it and it worked for them as well.

bchretien avatar Mar 07 '25 10:03 bchretien

can also confirm this dramatically improves the quality of the exported ply files, this needs to be in nerfstudio asap

CameronFraser avatar Apr 07 '25 15:04 CameronFraser

Sorry, I tested it with two setup:

  1. ns-train splatfacto-mcmc --pipeline.datamanager.batch-size 3 with gsplat 1.4.0
  2. ns-train splatfacto-mcmc --pipeline.datamanager.batch-size 3 with this PR of gsplat

But the result of 2nd setup is worse. Both quantitative (PSNR, LPIPS, SSIM) and visually qualitative.

So far, Tested it with garden, bicycle, and stump dataset from mip360.

1st setup: image

2nd setup: image

Dunno why it does not give good result from me. FYI. I have not tested it with native gsplat simple_trainer.py mcmc.

Additional info:

Ubuntu 22.04.3 LTS
Python 3.10.12
Torch 2.1.2+cu118
RTX 4090 on Runpod cloud environment
Nerfstudio commit d5bdd45b5bfd30e928e8b90d6735b2e38ae79a59

ichsan2895 avatar Apr 09 '25 13:04 ichsan2895

python3 examples/simple_trainer.py mcmc --batch-size 3:

1st = Gsplat 1.4.0 image

2nd = This PR of GSplat image

Additional benchmark of MCMC from @MrNeRF in this thread: https://github.com/nerfstudio-project/gsplat/pull/409. It uses batch-size 1. Untitled

ichsan2895 avatar Apr 10 '25 03:04 ichsan2895

@ichsan2895 thanks for the feedback! So maybe keep it disabled by default, so that users can enable it for scenes where this improves things? In my case, drone flights for which we look down and not far away (large splats are often located on the border of the scene but "spill" in the focused part).

bchretien avatar Apr 25 '25 06:04 bchretien