NerfBaselines

NerfBaselines is a framework for evaluating and comparing existing NeRF and 3DGS methods. Currently, most official implementations use different dataset loaders, evaluation protocols, and metrics, which renders benchmarking difficult. Therefore, this project aims to provide a unified interface for running and evaluating methods on different datasets in a consistent way using the same metrics. But instead of reimplementing the methods, we use the official implementations and wrap them so that they can be run easily using the same interface.

Please visit the project page to see the results of implemented methods on dataset benchmarks.

Project Page + Results  |  Paper

Getting started

Start by installing the nerfbaselines pip package on your host system.

pip install nerfbaselines

Now you can use the nerfbaselines cli to interact with NerfBaselines.

The next step is to choose the backend which will be used to install different methods. At the moment there are the following backends implemented:

• docker: Offers good isolation, requires docker (with NVIDIA container toolkit) to be installed and the user to have access to it (being in the docker user group).
• apptainer: Similar level of isolation as docker, but does not require the user to have privileged access.
• conda (default): Does not require docker/apptainer to be installed, but does not offer the same level of isolation and some methods require additional dependencies to be installed. Also, some methods are not implemented for this backend because they rely on dependencies not found on conda.
• python: Will run everything directly in the current environment. Everything needs to be installed in the environment for this backend to work.

The backend can be set as the --backend <backend> argument or using the NERFBASELINES_BACKEND environment variable.

Downloading data

For some datasets, e.g. Mip-NeRF 360, NerfStudio, Blender, or Tanks and Temples, the datasets can be downloaded automatically. You can specify the argument --data external://dataset/scene during training or download the dataset beforehand by running nerfbaselines download-dataset dataset/scene. Examples:

# Downloads the garden scene to the cache folder.
nerfbaselines download-dataset mipnerf360/garden

# Downloads all nerfstudio scenes to the cache folder.
nerfbaselines download-dataset nerfstudio

# Downloads kithen scene to folder kitchen
nerfbaselines download-dataset mipnerf360/kitchen -o kitchen

Training

To start the training, use the nerfbaselines train --method <method> --data <data> command. Use --help argument to learn about all implemented methods and supported features.

Rendering

The nerfbaselines render --checkpoint <checkpoint> command can be used to render images from a trained checkpoint. Again, use --help to learn about the arguments.

In order to render a camera trajectory (e.g., created using the interactive viewer), use the following command command:

nerfbaselines render-trajectory --checkpoint <checkpoint> --trajectory <trajectory> --output <output.mp4>

Interactive viewer

Given a trained checkpoint, the interactive viewer can be launched as follows:

nerfbaselines viewer --checkpoint <checkpoin> --data <dataset>

Even though the argument --data <dataset> is optional, it is recommended, as the camera poses are used to perform gravity alignment and rescaling for a better viewing experience. It also enables visualizing the input camera frustums.

Results

In this section, we present results of implemented methods on standard benchmark datasets. For detailed results, visit the project page: https://jkulhanek.com/nerfbaselines

Mip-NeRF 360

Mip-NeRF 360 is a collection of four indoor and five outdoor object-centric scenes. The camera trajectory is an orbit around the object with fixed elevation and radius. The test set takes each n-th frame of the trajectory as test views. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/mipnerf360

Blender

Blender (nerf-synthetic) is a synthetic dataset used to benchmark NeRF methods. It consists of 8 scenes of an object placed on a white background. Cameras are placed on a semi-sphere around the object. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/blender

Tanks and Temples

Tanks and Temples is a benchmark for image-based 3D reconstruction. The benchmark sequences were acquired outside the lab, in realistic conditions. Ground-truth data was captured using an industrial laser scanner. The benchmark includes both outdoor scenes and indoor environments. The dataset is split into three subsets: training, intermediate, and advanced. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/tanksandtemples

Reproducing results

Implementation status

Methods:

• [x] NerfStudio (Nerfacto)
• [x] Instant-NGP
• [x] Gaussian Splatting
• [x] Mip-Splatting
• [x] Gaussian Opacity Fields
• [x] Tetra-NeRF
• [x] Mip-NeRF 360
• [x] Zip-NeRF
• [x] CamP
• [x] TensoRF
• [x] K-Planes
• [ ] Nerf-W (open source reimplementation)
• [ ] NeRF on-the-go
• [ ] TRIPS
• [ ] Mip-NeRF
• [ ] NeRF

Datasets/features:

• [x] Mip-NeRF 360 dataset
• [x] Blender dataset
• [x] any COLMAP dataset
• [x] any NerfStudio dataset
• [x] LLFF dataset
• [x] Tanks and Temples dataset
• [x] Photo Tourism dataset and evaluation protocol
• [x] Bundler dataset format
• [x] automatic dataset download
• [x] interactive viewer and trajectory editor
• [x] undistorting images for methods that do not support complex camera models (Gaussian Splatting)
• [x] logging to tensorboard, wandb
• [ ] HDR images support
• [ ] RAW images support

Contributing

Contributions are very much welcome. Please open a PR with a dataset/method/feature that you want to contribute. The goal of this project is to slowly expand by implementing more and more methods.

Citation

If you use this project in your research, please cite the following paper:

@article{kulhanek2024nerfbaselines,
  title={NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods},
  author={Jonas Kulhanek and Torsten Sattler},
  year={2024},
  journal={arXiv},
}

License

This project is licensed under the MIT license Each implemented method is licensed under the license provided by the authors of the method. For the currently implemented methods, the following licenses apply:

• NerfStudio: Apache 2.0
• Instant-NGP: custom, research purposes only
• Gaussian-Splatting: custom, research purposes only
• Mip-Splatting: custom, research purposes only
• Gaussian Opacity Fields: custom, research purposes only
• Tetra-NeRF: MIT, Apache 2.0
• Mip-NeRF 360: Apache 2.0
• Zip-NeRF: Apache 2.0
• CamP: Apache 2.0

Acknowledgements

A big thanks to the authors of all implemented methods for the great work they have done. We would also like to thank the authors of NerfStudio, especially Brent Yi, for viser - a great framework powering the viewer. This work was supported by the Czech Science Foundation (GAČR) EXPRO (grant no. 23-07973X), the Grant Agency of the Czech Technical University in Prague (grant no. SGS24/095/OHK3/2T/13), and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).