Runtime error at the start of training

[lmattos-11]

Hi,

I am trying to train and evaluate the network for a small scale experiment. I have managed to get everything working up to training— setting up environment, rendering input image data, changing configurations, managing dependencies issues...

At the start of training in train.py I get the following error:

RuntimeError: cuda runtime error (8) : invalid device function at /opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp:405

I am running the training on a single NVIDIA GeForce GPU with 25GB memory usage. Driver Version: 510.60.02 and CUDA Version: 11.6. I suspect that this error comes from some incompatibility between my GPU CUDA version and the PyTorch and other packages versions set up by the environment provided on environment.yml

I was wondering if anyone or the authors have run into a similar issue or have any suggestions on how to manage this possible incompatibility.

I tried to install higher versions of PyTorch, torchvision, and install cudatoolkit on the environment but this lead to incompatibilities with other packages' versions specified in environment.yml. conda update --all also created dependency issues

[ngailapdi]

Hi @lmattos-11,

Apologize for the late response. I believe the issue can be fixed by installing the correct cudatoolkit version. The cudatoolkit in environment.yml is 10.2, please update this to match yours (version 11.6).