Future plans to upgrade nerfstudio

[HrsPythonix]

Hello, I am wondering if there is any plan to upgrade nerfstudio, eg. nerfacto model.

I see some recent research aiming to improve nerf's resolution like 4k_nerf and AligNeRF. Will them be added?

Also previously I tried use mask in training, but suffering from great performance issue, like huge memory cost in gpu, will this be fixed or optimized in the future?

[tancik]

We intend to both keep updating nerfacto and add other models as they are released (for example there is currently a PR for adding NeRFPlayer). @akristoffersen has an implementation of 4k nerf that he may be able to share, however it doesn't seem to provide any significant performance gains. No one is currently working on AligNerf, but we will accept PRs if people provide an implementation.

The mask issue is something we are aware of and intend to fix, just need to find someone, or sometime to work on it.

[akristoffersen]

@HrsPythonix I've got a branch alex/4k_nerf that is open right now that adds in some of the 4k_nerf techniques with the Nerfacto method. Though @tancik is right that it didn't seem to make anything significantly better.

That being said, there could be a way to increase the quality and/or I made a mistake in the hybrid implementation. Happy to take another look if there's interest.

[Dawars]

The way sdfstudio handles this is to only store the non-zero indices in memory. https://github.com/autonomousvision/sdfstudio/blob/master/nerfstudio/data/dataparsers/heritage_dataparser.py#L179 https://github.com/autonomousvision/sdfstudio/blob/master/nerfstudio/data/pixel_samplers.py#L108

Do you think this is a good way to handle it?

Notes:

• They load all of the masks in the DataParser upfront which is bad
• It might be better to store the zero indices since those might be fewer depending on application

[tancik]

A different solution is to pre-process/cache all of the data as rays at the beginning of training. We could then shuffle and then sample them like a more traditional ML pipeline.

[Dawars]

And then to support camera optimization we store the pixel location in the rays?

[tancik]

Yea, thats where it gets more challenging. Not sure on the top of my head.

[HrsPythonix]

ot a branch alex/4k_nerf that is open right now that adds in some of the 4k_nerf techniques with the Nerfacto method. Thoug

Thanks, I will take a look.

[Utopiah]

Not sure if that's the right issue for that but I'm on Ubuntu 22.04.1 LTS with NVIDIA 525.78.01 and Python 3.10 so I would assume something rather normal in that context. Unfortunately this comes with CUDA 12.0 which is now supported in pytorch https://github.com/pytorch/pytorch/pull/91118 so wondering what's the schedule in term of this kind of upgrades.