nerfstudio Unable to render to mp4 with `RuntimeError: stack expects a non-empty TensorList`

Describe the bug

Tried to render to MP4. got this error:

✅ Done loading checkpoint from outputs/lego_processed/nerfacto/2024-02-14_193442/nerfstudio_models/step-000029999.ckpt
Traceback (most recent call last):
  File "/usr/local/bin/ns-render", line 8, in <module>
    sys.exit(entrypoint())
  File "/usr/local/lib/python3.10/dist-packages/nerfstudio/scripts/render.py", line 896, in entrypoint
    tyro.cli(Commands).main()
  File "/usr/local/lib/python3.10/dist-packages/nerfstudio/scripts/render.py", line 456, in main
    camera_path = get_path_from_json(camera_path)
  File "/usr/local/lib/python3.10/dist-packages/nerfstudio/cameras/camera_paths.py", line 178, in get_path_from_json
    camera_to_worlds = torch.stack(c2ws, dim=0)
RuntimeError: stack expects a non-empty TensorList
root@3eada9a39237:/workspace# ns-render camera-path --load-config outputs/lego_processed/nerfacto/2024-02-14_193442/config.yml --camera-path-filename /workspace/lego_processed/camera_paths/2024-02-14-19-34-49.json --output-path renders/lego_processed/2024-02-14-19-34-49.mp4

To Reproduce

Download images for training. https://files.extrastatic.dev/extrastatic/Photos-001.zip. Unzip into a directory called lego
ns-process-data images --data lego/ --output-dir lego_processed
ns-train nerfacto --data lego_processed/
When completed, try to run render command: ns-render camera-path --load-config outputs/lego_processed/nerfacto/2024-02-14_193442/config.yml --camera-path-filename /workspace/lego_processed/camera_paths/2024-02-14-19-34-49.json --output-path renders/lego_processed/2024-02-14-19-34-49.mp4

Expected behavior

It should create the mp4 file.

Additional context

Using docker:

docker build \
    --build-arg CUDA_VERSION=11.8.0 \
    --build-arg CUDA_ARCHITECTURES=86 \
    --build-arg OS_VERSION=22.04 \
    --tag nerfstudio-86:0.0.1 \
    --file Dockerfile .

Inside container nvidia-smi:

# nvidia-smi 
Wed Feb 14 20:39:13 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.02              Driver Version: 545.29.02    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        Off | 00000000:0A:00.0 Off |                  N/A |
|  0%   41C    P8              23W / 370W |      5MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

colmap -h

# colmap -h
COLMAP 3.8 -- Structure-from-Motion and Multi-View Stereo
              (Commit 43de802 on 2023-01-31 with CUDA)

Usage:
  colmap [command] [options]

Documentation:
  https://colmap.github.io/

Example usage:
  colmap help [ -h, --help ]
  colmap gui
  colmap gui -h [ --help ]
  colmap automatic_reconstructor -h [ --help ]
  colmap automatic_reconstructor --image_path IMAGES --workspace_path WORKSPACE
  colmap feature_extractor --image_path IMAGES --database_path DATABASE
  colmap exhaustive_matcher --database_path DATABASE
  colmap mapper --image_path IMAGES --database_path DATABASE --output_path MODEL
  ...

Available commands:
  help
  gui
  automatic_reconstructor
  bundle_adjuster
  color_extractor
  database_cleaner
  database_creator
  database_merger
  delaunay_mesher
  exhaustive_matcher
  feature_extractor
  feature_importer
  hierarchical_mapper
  image_deleter
  image_filterer
  image_rectifier
  image_registrator
  image_undistorter
  image_undistorter_standalone
  mapper
  matches_importer
  model_aligner
  model_analyzer
  model_comparer
  model_converter
  model_cropper
  model_merger
  model_orientation_aligner
  model_splitter
  model_transformer
  patch_match_stereo
  point_filtering
  point_triangulator
  poisson_mesher
  project_generator
  rig_bundle_adjuster
  sequential_matcher
  spatial_matcher
  stereo_fusion
  transitive_matcher
  vocab_tree_builder
  vocab_tree_matcher
  vocab_tree_retriever

Last commit from git log:

commit 4f798b23f6c65ef2970145901a0251b61ec8a447 (HEAD -> main, origin/main, origin/HEAD)
Author: Sebastiaan <[email protected]>
Date:   Tue Feb 13 19:09:59 2024 +0100
...

Feb 14 '24 20:02 xrd

I've noticed that this issue also always appears to happen to me when I only have 1 keyframe trying to render a single image.

The moment I add another camera keyframe in the path, multiple images will render.

I assume it may not be exactly the same issue since you were trying to create a video (which would usually have multiple frames), but at least this is a way to provide a minimum reproducible example that seems to always result in the error.

{
  "default_fov": 75,
  "default_transition_sec": 2,
  "keyframes": [
    {
      "matrix": [
        0.48770182459400074, -0.6865172221850452, -0.5392967957729877,
        -0.4507239373890963, -0.8727777603899364, -0.369160257136332,
        -0.31934258331729526, -0.2323788609443143, 0.020147239424029684,
        0.6264302101546237, -0.7792170946207637, -0.7966745989802786,
        0, 0, 0, 1
      ],
      "fov": 75,
      "aspect": 1.7777777777777777,
      "override_transition_enabled": false,
      "override_transition_sec": null
    }
  ],
  "camera_type": "perspective",
  "render_height": 1080,
  "render_width": 1920,
  "fps": 30,
  "seconds": 0,
  "is_cycle": false,
  "smoothness_value": 0,
  "camera_path": []
}

E.g. as we can see the camera path list is empty. And in the code the "camera_to_world" key is supposed to be read and added to c2ws, which is later used in the line erroring. It fails because c2ws is empty since there was no camera in the empty "camera_path".

https://github.com/nerfstudio-project/nerfstudio/blob/d1fc2ee33863071aa03c6679595d554d67246258/nerfstudio/cameras/camera_paths.py#L149-L155

https://github.com/nerfstudio-project/nerfstudio/blob/d1fc2ee33863071aa03c6679595d554d67246258/nerfstudio/cameras/camera_paths.py#L178