I have my dataset in the format that 3D Gaussian Splatting and NerfStudio accept. That is a folder with an images subfolder and camera params of those images saved in a json file. How do I change them in the format that PixelSplat accepts?

[Shahid1Malik]

i have my dataset in the format that 3D Gaussian Splatting and NerfStudio accepts. That is an image folder with camera parameters of those images saved in a json file. How do I change them in the format that PixelSplat accepts?

[donydchen]

Hi @Shahid1Malik, sorry for the late reply, I've been busy with several deadlines previously.

To run on your own dataset, the straightforward way is to convert your data into similar torch files so that you can easily reuse the data loader.

To convert data to torch files, refer to how we converted the DTU dataset at  https://github.com/donydchen/mvsplat/blob/main/src/scripts/convert_dtu.py. We have also had more discussions regarding this convert script. Please refer to https://github.com/donydchen/mvsplat/issues/28.

If the camera parameters are in the format that 3D Gaussian Splatting and NerfStudio accept, my best guess is that they are obtained via COLMAP. If so, you can refer to https://github.com/donydchen/mvsplat/issues/1 for instructions on converting the COLMAP data.

After successfully building the torch files, the most important thing is to confirm that the camera parameters are correctly formatted. You can find more related discussions at https://github.com/donydchen/mvsplat/issues/23#issuecomment-2085190160.

[Shahid1Malik]

i have converted a subset of real10k using your convert.py script. However the torch file generated is 47 MB. when i feed it for evaluation, the testing doesnt work, it appears like it does not find or accept the torch file. What could be the problem here

[donydchen]

Hi @Shahid1Malik, several reasons can lead to these issues. Below, I provide some potential solutions to help you debug.

• make sure the torch data is correctly loaded; you can do it by printing the self.chunks right after https://github.com/donydchen/mvsplat/blob/main/src/dataset/dataset_re10k.py#L83

• if you are running the testing, make sure that the scenes exist in the eval index file (e.g., assets/evaluation_index_re10k.json); you can confirm it at https://github.com/donydchen/mvsplat/blob/main/src/dataset/view_sampler/view_sampler_evaluation.py#L54

[XLR-man]

I have changed my custom datasets which are colmap data format to .torch files.

And when i test it, it got an error:

Traceback (most recent call last):
  File "/data/xielangren/project/mvsplat/src/main.py", line 154, in train
    trainer.test(
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 754, in test
    return call._call_and_handle_interrupt(
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in _test_impl
    results = self._run(model, ckpt_path=ckpt_path)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
    results = self._run_stage()
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1026, in _run_stage
    return self._evaluation_loop.run()
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 135, in run
    self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 396, in _evaluation_step
    output = call._call_strategy_hook(trainer, hook_name, *step_args)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 425, in test_step
    return self.lightning_module.test_step(*args, **kwargs)
  File "/data/xielangren/project/mvsplat/src/model/model_wrapper.py", line 186, in test_step
    gaussians = self.encoder(
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/jaxtyping/_decorator.py", line 450, in wrapped_fn
    out = fn(*args, **kwargs)
  File "/data/xielangren/project/mvsplat/src/model/encoder/encoder_costvolume.py", line 202, in forward
    gaussians = self.gaussian_adapter.forward(
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/jaxtyping/_decorator.py", line 450, in wrapped_fn
    out = fn(*args, **kwargs)
  File "/data/xielangren/project/mvsplat/src/model/encoder/common/gaussian_adapter.py", line 90, in forward
    harmonics=rotate_sh(sh, c2w_rotations[..., None, :, :]),
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/jaxtyping/_decorator.py", line 450, in wrapped_fn
    out = fn(*args, **kwargs)
  File "/data/xielangren/project/mvsplat/src/misc/sh_rotation.py", line 18, in rotate_sh
    alpha, beta, gamma = matrix_to_angles(rotations)
  File "/data/xielangren/miniconda3/envs/mvsplat/lib/python3.10/site-packages/e3nn/o3/_rotation.py", line 404, in matrix_to_angles
    assert torch.allclose(torch.det(R), R.new_tensor(1))
AssertionError

how can i solve it?