nerfstudio
nerfstudio copied to clipboard
nerfstudio-project
nerfacto overfitting on blender scenes?
Describe the bug When I train nerfacto on any of the blender scenes, it strongly overfits the input views: render is OK from the input viewpoints, but terrible from any other viewpoint.
The nerfstudio datasets work fine, as well as my custom datasets from video, maybe because the viewpoints are much closer?
When I do a custom dataset from wide-baseline images, I get kind of the same behavior as with the blender datasets (image overfit).
Is that expected?
I'd like a working train/test configuration to work on testing improvements of nerfstudio.
If that matters, I am training on compute arch 70 (p3.2xlarge EC2 instance), so I get the following messages:
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
To Reproduce Steps to reproduce the behavior:
viewer_opts="--vis viewer --viewer.websocket-port=7007"
ns-train nerfacto $viewer_opts blender-data --data data/blender/lego
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Input viewpoint:
rotate slightly:
rotate more:
Additional context
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.08 Driver Version: 510.73.08 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000000:00:1E.0 Off | 0 |
| N/A 41C P0 88W / 300W | 5987MiB / 16384MiB | 42% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Nerfacto was designed to work well on real scenes (which in practice hurts performance on blender scenes). Take a look at #806 for some extra arguments for a better blender configuration using nerfacto.
Maybe we should aply those changes to the benchmarking scripts, or (better option IMHO) use a real-life dataset for benchmarking (T&T?).
Yea, we really should have better benchmarking. Hopefully something we can improve going forward. One open challenge is how to properly evaluate methods on real data. Methods like nerfacto build in tricks to handle differences in training image exposure and slightly misaligned poses. As a result it is unclear what exposure or pose you should use to render an eval image. There are some tricks people use like trying to match the appearance on half an image, can compute the metrics on the other half. Ideally we would have more principled methods.
what exposure or pose you should use
Indeed, evaluation metrics significantly are lower when camera optimizations are on. Matching half of image to align for target exposure/appearance embedding should not be too complicated, but Do you think running a inverse rendering to solve for "new" pose of eval image using only half of the image is possible? Especially during early stages of the training?
