When does camera optimizer make things worse? Can it be detected?

[f-dy]

Posting this to trigger a discussion:

I've seen cases where the camera optimizer (with default parameters) makes things worse. It is known to degrade results on synthetic datasets (eg blender lego), but I've seen it fail even on real images, especially images with low texture but strong edges: the output was way more blurry than without camera optimization. Dividing the camera optimizer learning rate by 10 (from 6e-4 to 6e-5) mitigated that issue in my case, but I wonder if we're missing some guardrails in nerfstudio: the camera optimizer should never make things worse. Even the default LR value (6e-4) makes me think it was not trivial to tune that parameter.

Have other users witnessed a similar behavior?

It seems that, when it goes wrong, that happens during the very first iterations. To check for that, I trained without camera optimization, which gave me a good result (camera poses from SfM), and then loaded that checkpoint to run a second training with camera optimization and default LR (maybe it's not supposed to work?). After just a few iterations, everything became very blurry (sorry, I can't show those results), and it never fully recovered from that. My guess is that this could be reproducible with blender-lego.

Could there be an optimization strategy such that the result is never worse than without camera pose optimization, or should we always run without and with optimization (or maybe with two LR values) and take the best result?

Does it make sense to fully enable camera optimization at the first iteration? Adam has not yet learned anything about the gradients at this point, so it may be sending camera poses very far away from their initial (and potentially almost-correct) position.

May be related:

• https://github.com/nerfstudio-project/nerfstudio/issues/1025
• https://github.com/nerfstudio-project/nerfstudio/issues/1017

[tancik]

If it is an issue in the first few iterations, it might be worth experimenting with a scheduler on the optimizer.

[punsii]

I have found this to be a big problem when training nerfacto(-big/huge) on various datasets. Colmap poses look good but after the first few iterations the camera poses are considerably altered and don't line up anymore. For anyone looking for an immediate solution: With most models you can set "--optimizers.camera-opt.optimizer.lr" in the ns-train command to a value like 0.0001 or lower.

To solve this problem i would be in favor of reducing the camera optimizer learning rate a lot by default and add a flag that indicates that the poses are bad / have not been found using a good sfm tool, which then increases it again. Maybe "--improve-initial-camera-poses"? Not sure about the name.

[f-dy]

add a flag that indicates that the poses are bad / have not been found using a good sfm tool

I don't understand that part. Could you please reformulate or expand your idea?