Support MCMC Strategy in Splatfacto

[yufengzh]

This PR adds the MCMC strategy for Splatfacto. To use, specify the following in the CLI argument:

ns-train splatfacto --pipeline.model.strategy=mcmc

[abrahamezzeddine]

Hello,

I just tried this and it seems to work. I am however wondering about the huge splats being created and large splats going straight through the model, which creates the illusion of hue artifacts inside the area of interest. See image below;

Is that an inherent problem with mcmc in general? Another example with depth map to show how huge splats are just "going through" the model

They are several magnitudes larger than sparse cloud itself.

[kerrj]

Thanks for the PR! I believe this is missing the opacity + scale regularization in the original MCMC paper, do you think you could add those? that might be a reason for these large gaussians during training

[yufengzh]

Does gsplat have those regularization terms implemented or not yet?

[CameronFraser]

@yufengzh they are not part of gsplat, there is an example here for how to implement: https://github.com/nerfstudio-project/gsplat/blob/main/examples/simple_trainer.py#L661

  # regularizations
  if cfg.opacity_reg > 0.0:
      loss = (
          loss
          + cfg.opacity_reg
          * torch.abs(torch.sigmoid(self.splats["opacities"])).mean()
      )
  if cfg.scale_reg > 0.0:
      loss = (
          loss
          + cfg.scale_reg * torch.abs(torch.exp(self.splats["scales"])).mean()
      )

[CameronFraser]

@kerrj I see there is a use_scale_regularization config here: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/splatfacto.py#L195 and then there is a scale_reg loss calculated here: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/splatfacto.py#L720 but doesn't this need to be applied to the main loss like in gsplat? or does that happen elsewhere?

[kerrj]

The existing scale_reg loss is actually a different penalty from the PhysGauss paper, which is unrelated to MCMC. (it's added to the main loss a few lines below when it's added to the loss dict).

@yufengzh gsplat only provides the rasterization interface/splitting strategy, since the loss calculation is done inside splatfacto it'll have to be added here. The loss is pretty straightforward if i recall, it's just penalizing the mean of all scales, and the mean of all opacities (theres an equation in the paper describing it)

[CameronFraser]

@yufengzh are you still working on this PR? excited to see it get merged. Do you need any help with the regularization terms?

[Ben-Mack]

@liruilong940607 @jb-ye Can you give some review to this PR? Are there some thing needed to do before merge? I see many people are really looking forward for MCMC to be implemented into NerfStudio. Hope you guys can review this PR soon.

[CameronFraser]

@Ben-Mack the PR is not complete, the regularlization terms need to be added

[kerrj]

Closing this since #3548 is being merged in which implements MCMC!