auto_orient_poses shift object out of the scene

[dozeri83]

Hi, first of all great job! your tool looks very promising. I am trying to experiment with it.

The first thing I saw is that you are shifting the cameras automatically according the camera poses. sometimes the object in interest is out of the cameras poses bounding box.

For example , look at the blender "chair" data set - auto orienting causes the chair to be out of the scene.

I would suggest one of the following:

1. dont auto orient by default, only when user demand it with a flag. (I for example use tiepoint from colmap for the orientation as preprocess step)

2. when you auto orient - you should auto orient the scene box (aabb) as well.

[tancik]

Hi, I'd like to dig into your suggestions a little more.

1. I am unfamiliar with the tiepoint in colmap, do you have documentation?
2. Currently the aabb should be updated with the auto-orient. Are you seeing something else?

If you have screenshots, that could also be helpful.

[dozeri83]

1. by tiepoints I mean the sparse points cloud that is generated from the camera alignment process points3D.txt/bin https://colmap.github.io/format.html This point cloud provides useful information since it represents the actual scene that you want to render. Using it you can align the data such that the cloud will fit in the aabb box.

2. Looking at - https://github.com/nerfstudio-project/nerfstudio/blob/345ef84be6f709fae712c2e6b2600463cdb05aae/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#L117 It does not look like you align the aabb, you just align the poses.

look for example at a screen shot of "chair" data set: The chair is located below the origin.

To conclude, I thing that aligning/not aligning of the camera poses automatically should be passed as a flag, since it is very difficult to determine where the actual scene is.

[tobias-kirschstein]

I can confirm that the auto orientation is a problem for us as well. Cameras are moved to the origin where they do not belong in our case. This can cause the object of interest to be out of the scene box if an AABB contraction type is used. This in turn causes CUDA out of memory errors since the num_samples_per_batch coming out of the ray marching visibility check will be a very small number causing the dynamic_num_rays_per_batch in the DynamicBatchPipeline to sky-rocket...

https://github.com/nerfstudio-project/nerfstudio/blob/52e2c61e6b96ab283d3bb1b70b634f4fb2a1e0a0/nerfstudio/pipelines/dynamic_batch.py#L73

From my understanding, the auto-orientation only makes sense if the cameras circle around the object of interest. An assumption that is not always true. I would turn it off by default or provide an additional orientation method that moves the cameras such that they look at the origin

[tancik]

Adding some additional options for loading the dataset. Hopefully they will be useful.

[dozeri83]

can you give an example of how to use it in command line?

[tancik]

If you don't want our dataparser to change the poses at all, use the following,

ns-train nerfacto --data DATA nerfstudio-data --orientation-method none --center-poses False --auto-scale-poses False