Workaround for torch.linalg lazy init for multi-GPU setup?

[Nicholas-Autio-Mitchell]

❓ Questions/Help/Support

I am new to the ignite library, and am trying to run multi-GPU (single node) training that is configured in this repo, and here is the specific entrypoint I am running where ignite.distributed.Parallel is called.

The error I am facing seems quite generic and that suggests a setup issue:

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/user/.../lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/home/user/.../lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/.../BehindTheScenes/models/bts/trainer.py", line 115, in forward
    to_base_pose = torch.inverse(poses[:, :1, :, :])
RuntimeError: lazy wrapper should be called at most once

I found this error and explanation on torch repo: https://github.com/pytorch/pytorch/issues/90613. It suggests newer CUDA versions can handle it better and provides a hack: calling torch.inverse(...) at the entrpoint to force the lazy initialisation of torch.linalg to happen. This doesn't fix the problem for :(

I don't want to get into the details of the other repo with this question, just first wanted to ask if there is a known workaround when using the ignite library (apart from upgrading CUDA)?

Thanks in advance for any help!

---

Version info

The full environment given by the above repo: https://github.com/Brummi/BehindTheScenes/blob/main/environment.yml

• OS: Ubuntu 18.04.5 LTS
• CUDA Driver Version: 470.182.03
• Python env:

  cuda                      11.6.1                        0    nvidia
  cuda-toolkit              11.6.1                        0    nvidia
  cuda-tools                11.6.1                        0    nvidia
  python                    3.10.12              h955ad1f_0
  pytorch                   1.13.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
  pytorch-cuda              11.6                 h867d48c_1    pytorch
  pytorch-ignite            0.4.12                   pypi_0    pypi
  

[vfdev-5]

@Nicholas-Autio-Mitchell seems like the original repository is using single gpu training with a single gpu (if they do not encounter any similar issues). In your case, most probably you have multiple gpus and you can launch the same training by exposing only a single cuda device, for example:

CUDA_VISIBLE_DEVICES=0 python train.py -cn exp_kitti_360

If you would like to train on multiple gpus using DDP, you can try it using (I checked quickly the code and it looks like it should be able to parallelize) + add "backend: 'nccl'" into the config yaml file:

torchrun --nproc_per_node=N train.py -cn exp_kitti_360

Hope this helps

[Nicholas-Autio-Mitchell]

Thanks for your answer @vfdev-5.

Using only a single GPU does work; I had already been training with a single GPU using CUDA_VISIBLE_DEVICES as you suggested.

I had not tried using torchrun explicitly (just python train.py ...). That does solve the lazy-loading problem, but raises a different issue when using >1 GPUs:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by
making sure all `forward` function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 1: 159 160 183 184 185 186 187 188

I have managed to find the unused model parameters and remove them (final layers of pre-trained backbone).

Is there any way to simply allow unused parameters?

[vfdev-5]

@Nicholas-Autio-Mitchell I think you can pass find_unused_parameters=True to idist.auto_model(..., find_unused_parameters=True) and it should be working.

[Nicholas-Autio-Mitchell]

Yes, that did work fine - as mentioned, I found and removed them so distributed training is working.

I was wondering if it is possible to simply allow unused parameters? The setup is quite complex and I don't want to find and remove parameters for all training configurations and dataset variants.