nerfstudio
nerfstudio copied to clipboard
nerfstudio-project
Add temporal distortion and D-NeRF model
Rather than recreate an entire new model, blend Dynamic-NeRF into the existing vanilla nerf model. It also adds a temporal distortion which can be applied in the middle of the Vanilla NeRF pipeline.
Not sure whether the best way to incorporate this approach is to use this, or if it makes sense to incorporate a way to add distortion directly into each model. This should be possible to add into TensoRF directly without modification (and also so I don't have to wait so long for convergence of the model), and am curious what the results of that would be like. If wanted, I can make changes to add a flag for that instead of creating an entirely separate model.
This should be ready for review! I know y'all busy but just wanted to bump it in case you missed it. I should also probably join the discord or smth so I can directly ask what the desired outcome state of this kind of thing would be.
I do need to check convergence, but any comments on the code structure would be appreciated.
This should be ready for review! I know y'all busy but just wanted to bump it in case you missed it. I should also probably join the discord or smth so I can directly ask what the desired outcome state of this kind of thing would be.
Feel free to ping me on the discord server.
This should be ready for review again, but really I should post pictures of convergence first.
Picture of it slowly converging (albeit artifacts but I'm not sure if they're present in the original approach)
I also noticed if I zoom out very far, nothing will appear because outside of the cameras' viewpoints the temporal distortion is not well defined. I may include a component for AABB checks unless it would be outside of the scope.
Image from Tensorboard:
(Ground Truth, Coarse, Fine)
Copy-Paste from D-NeRF paper
Eval Images Metrics/fine_psnr of current implementations
Currently the results of this implementation on the Lego D-NeRF dataset appear to be better, not sure exactly why, but could be because of the fourier features instead of positional encoding.
@tancik I still don't have permissions to do so, I think it's probably collaborators on the repo or members of the org who have permissions
