Multi-scene neural fields

[akshay-krishnan]

Is your feature request related to a problem? Please describe.

An example of a multi-scene field: https://nesf3d.github.io/ Such a field/network will be trained on rays and ground truth from multiple scenes. The network will need to be re-parameterized for each training scene. This could involve reloading weights of a subnetwork, passing in a latent code, or other data.

Describe the solution you'd like

A way to re-parameterize models for the specific scene before training, by fetching scene-specific parameters from the dataset. This could be done by the before-step training callbacks that nerfstudio supports, assuming that each step (batch) will only contain data from the specific scene. But I am not sure how to pass this data (scene-specific parameters that are per-batch instead of per-batch-item) from a dataloader to the model. The current callback arguments seem to be config settings.

Describe alternatives you've considered Setting up my own pipeline for this :)

This may be expecting too much from nerfstudio, and I am not sure if you plan to support such models, but I wanted to check anyway. I see some documentation in the data manager that suggests variable scenes would be supported, but I am not sure if this is the same as the use case I am implying.

I am happy to help with implementing this, but I would like your thoughts/suggestions first.