CUDA call was originally invoked at:

[Ljango]

trainer = Trainer(args= args,model=model,logger=logger,criterion=BCEWithLogLoss(),optimizer=optimizer,

File "/data2/zhengshaojie/Cog_CLS/new_bert/Bert-Multi-Label-Text-Classification/pybert/train/trainer.py", line 27, in init self.model, self.device = model_device(n_gpu = args.n_gpu, model=self.model) File "/data2/zhengshaojie/Cog_CLS/new_bert/Bert-Multi-Label-Text-Classification/pybert/common/tools.py", line 101, in model_device model = model.to(device) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1682, in to return super().to(*args, **kwargs) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to return self._apply(convert) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply module._apply(fn) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply module._apply(fn) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply module._apply(fn) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply param_applied = fn(param) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/cuda/init.py", line 264, in _lazy_init raise DeferredCudaCallError(msg) from e torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "../aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch.

CUDA call was originally invoked at:

[' File "run_bert.py", line 1, in \n import torch\n', ' File "", line 991, in _find_and_load\n', ' File "", line 975, in _find_and_load_unlocked\n', ' File "", line 671, in _load_unlocked\n', ' File "", line 843, in exec_module\n', ' File "", line 219, in _call_with_frames_removed\n', ' File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/init.py", line 1146, in \n _C._initExtension(manager_path())\n', ' File "", line 991, in _find_and_load\n', ' File "", line 975, in _find_and_load_unlocked\n', ' File "", line 671, in _load_unlocked\n', ' File "", line 843, in exec_module\n', ' File "", line 219, in _call_with_frames_removed\n', ' File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/cuda/init.py", line 197, in \n _lazy_call(_check_capability)\n', ' File "/data2/zhengshaojie/miniconda3/envs/GAL/lib/python3.8/site-packages/torch/cuda/init.py", line 195, in _lazy_call\n _queued_calls.append((callable, traceback.format_stack()))\n']