When does camera optimizer make things worse? Can it be detected?

[f-dy]

Posting this to trigger a discussion:

I've seen cases where the camera optimizer (with default parameters) makes things worse. It is known to degrade results on synthetic datasets (eg blender lego), but I've seen it fail even on real images, especially images with low texture but strong edges: the output was way more blurry than without camera optimization. Dividing the camera optimizer learning rate by 10 (from 6e-4 to 6e-5) mitigated that issue in my case, but I wonder if we're missing some guardrails in nerfstudio: the camera optimizer should never make things worse. Even the default LR value (6e-4) makes me think it was not trivial to tune that parameter.

Have other users witnessed a similar behavior?

It seems that, when it goes wrong, that happens during the very first iterations. To check for that, I trained without camera optimization, which gave me a good result (camera poses from SfM), and then loaded that checkpoint to run a second training with camera optimization and default LR (maybe it's not supposed to work?). After just a few iterations, everything became very blurry (sorry, I can't show those results), and it never fully recovered from that. My guess is that this could be reproducible with blender-lego.

Could there be an optimization strategy such that the result is never worse than without camera pose optimization, or should we always run without and with optimization (or maybe with two LR values) and take the best result?

Does it make sense to fully enable camera optimization at the first iteration? Adam has not yet learned anything about the gradients at this point, so it may be sending camera poses very far away from their initial (and potentially almost-correct) position.

May be related:

• https://github.com/nerfstudio-project/nerfstudio/issues/1025
• https://github.com/nerfstudio-project/nerfstudio/issues/1017