gsplat icon indicating copy to clipboard operation
gsplat copied to clipboard

Benchmark script (gsplat evaluation metrics)

Open maturk opened this issue 1 year ago • 0 comments

Note, this PR does not have to be merged. It can stay as a PR or if people find it useful, it can be merged.

As mentioned in #323, here is a benchmark script to run various configurations of the gsplat library. This script was used to reproduce the metrics shown in https://github.com/nerfstudio-project/gsplat/commit/1cc3d22d2e5455988b06e1fc8b7c00e3ea5f32f2.

This script is inspired by @niujinshuchong work.

This script is usable for multi-gpu batch runs.

It makes a queue of jobs of the form [(config_0, scene_0), (config_0, scene_1), ... (config_0, scene_n), (config_1, scene_0), ...] and available GPUs take a config from the queue and processes it.

Configs can be defined easily. E.g.

baseline_config = BenchmarkConfig()
mcmc_config = BenchmarkConfig(
    trainer="simple_trainer_mcmc.py",
    result_dir="results/mcmc",
)
# Configs to run in batch
configs_to_run = [baseline_config, mcmc_config]

maturk avatar Aug 08 '24 18:08 maturk