Segmentation Fault during launch Training on Nerfies

[AndreyKrotkikh]

Hi, tried to reproduce training process but got segmentation fault, logs are below:

python dgmesh/train.py --config dgmesh/configs/nerfies/tail.yaml Configuration loaded from dgmesh/configs/nerfies/tail.yaml Optimizing dgmesh/outputs/nerfies/tail Output folder: dgmesh/outputs/nerfies/tail/tail-2024-07-17_09-24-58 /opt/conda/envs/dg-mesh/lib/python3.9/site-packages/torch/functional.py:512: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905969824/work/aten/src/ATen/native/TensorShape.cpp:3587.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] /opt/conda/envs/dg-mesh/lib/python3.9/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( [17/07 09:24:59] /home/ubuntu/DG-Mesh/dgmesh/data/nerfies/tail [17/07 09:24:59] /home/ubuntu/DG-Mesh/dgmesh/data/nerfies/tail/dataset.json [17/07 09:24:59] [17/07 09:24:59] Assuming Nerfies data set! [17/07 09:24:59] Reading Nerfies Info [17/07 09:24:59] [17/07 09:25:16] Loading Training Cameras [17/07 09:25:16] Loading Test Cameras [17/07 09:25:18] Number of points at initialisation : 29192 [17/07 09:25:19] Training progress: 0%| | 0/25001 [00:00<?, ?it/s] Segmentation fault (core dumped)

Maybe you already know how to avoid or fix this?