trackformer
trackformer copied to clipboard
timmeinhardt
Problems on running VisDrone2019 dataset
Am suffering from CUDA Error during I was running Trackformer on VisDrone2019 dataset on a server with 3 RTX3090 GPUs:
Start training
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim = 0 && idx_dim = 0 && idx_dim = 0 && idx_dim
train(args)
File "src/train.py", line 284, in train
visualizers['train'], args)
File "/root/autodl-tmp/trackformer/src/trackformer/engine.py", line 128, in train_one_epoch
loss_dict = criterion(outputs, targets)
File "/root/miniconda3/envs/tf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 406, in forward
losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes))
File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 380, in get_loss
return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs)
File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 252, in loss_labels_focal
losses['class_error'] = 100 - accuracy(src_logits[idx], target_classes_o)[0]
RuntimeError: CUDA error: device-side assert triggered
My conda environment is listed as below:
_libgcc_mutex 0.1 main https://mirrors.ustc.edu.cn/anaconda/pkgs/main _openmp_mutex 5.1 1_gnu https://mirrors.ustc.edu.cn/anaconda/pkgs/main argon2-cffi 20.1.0 pypi_0 pypi astroid 2.4.2 pypi_0 pypi async-generator 1.10 pypi_0 pypi attrs 19.3.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi bleach 3.2.3 pypi_0 pypi ca-certificates 2023.5.7 hbcca054_0 conda-forge certifi 2020.4.5.2 pypi_0 pypi cffi 1.14.4 pypi_0 pypi chardet 3.0.4 pypi_0 pypi cloudpickle 1.6.0 pypi_0 pypi colorama 0.4.3 pypi_0 pypi cycler 0.10.0 pypi_0 pypi cython 0.29.20 pypi_0 pypi decorator 4.4.2 pypi_0 pypi defusedxml 0.6.0 pypi_0 pypi docopt 0.6.2 pypi_0 pypi entrypoints 0.3 pypi_0 pypi filelock 3.0.12 pypi_0 pypi flake8 3.8.3 pypi_0 pypi flake8-import-order 0.18.1 pypi_0 pypi future 0.18.2 pypi_0 pypi gdown 3.12.2 pypi_0 pypi gitdb 4.0.5 pypi_0 pypi gitpython 3.1.3 pypi_0 pypi idna 2.9 pypi_0 pypi imageio 2.8.0 pypi_0 pypi importlib-metadata 1.6.1 pypi_0 pypi ipykernel 5.4.3 pypi_0 pypi ipython 7.19.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 7.6.3 pypi_0 pypi isort 5.6.4 pypi_0 pypi jedi 0.18.0 pypi_0 pypi jinja2 2.11.2 pypi_0 pypi jsonpatch 1.25 pypi_0 pypi jsonpickle 1.4.1 pypi_0 pypi jsonpointer 2.0 pypi_0 pypi jsonschema 3.2.0 pypi_0 pypi jupyter 1.0.0 pypi_0 pypi jupyter-client 6.1.11 pypi_0 pypi jupyter-console 6.2.0 pypi_0 pypi jupyter-core 4.7.0 pypi_0 pypi jupyterlab-pygments 0.1.2 pypi_0 pypi jupyterlab-widgets 1.0.0 pypi_0 pypi kiwisolver 1.2.0 pypi_0 pypi lap 0.4.0 py37hfb7772e_1005 conda-forge lapsolver 1.1.0 pypi_0 pypi lazy-object-proxy 1.4.3 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main libblas 3.9.0 15_linux64_openblas conda-forge libcblas 3.9.0 15_linux64_openblas conda-forge libffi 3.4.4 h6a678d5_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main libgcc-ng 11.2.0 h1234567_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main libgfortran-ng 13.1.0 h69a702a_0 conda-forge libgfortran5 13.1.0 h15d22d2_0 conda-forge libgomp 11.2.0 h1234567_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main liblapack 3.9.0 15_linux64_openblas conda-forge libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge libstdcxx-ng 11.2.0 h1234567_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main markupsafe 1.1.1 pypi_0 pypi matplotlib 3.2.1 pypi_0 pypi mccabe 0.6.1 pypi_0 pypi mistune 0.8.4 pypi_0 pypi more-itertools 8.4.0 pypi_0 pypi motmetrics 1.2.0 pypi_0 pypi multiscaledeformableattention 1.0 pypi_0 pypi munch 2.5.0 pypi_0 pypi nbclient 0.5.1 pypi_0 pypi nbconvert 6.0.7 pypi_0 pypi nbformat 5.1.2 pypi_0 pypi ncurses 6.4 h6a678d5_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main nest-asyncio 1.5.1 pypi_0 pypi networkx 2.4 pypi_0 pypi ninja 1.10.0.post2 pypi_0 pypi notebook 6.2.0 pypi_0 pypi numpy 1.18.5 pypi_0 pypi opencv-python 4.2.0.34 pypi_0 pypi openssl 1.1.1u h7f8727e_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main packaging 20.4 pypi_0 pypi pandas 1.0.5 pypi_0 pypi pandocfilters 1.4.3 pypi_0 pypi parso 0.8.1 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 7.1.2 pypi_0 pypi pip 22.3.1 py37h06a4308_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main pluggy 0.13.1 pypi_0 pypi prometheus-client 0.9.0 pypi_0 pypi prompt-toolkit 3.0.14 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi py 1.8.2 pypi_0 pypi py-cpuinfo 6.0.0 pypi_0 pypi pyaml 20.4.0 pypi_0 pypi pycocotools 2.0 pypi_0 pypi pycodestyle 2.6.0 pypi_0 pypi pycparser 2.20 pypi_0 pypi pyflakes 2.2.0 pypi_0 pypi pygments 2.7.4 pypi_0 pypi pylint 2.6.0 pypi_0 pypi pyparsing 2.4.7 pypi_0 pypi pyrsistent 0.17.3 pypi_0 pypi pysocks 1.7.1 pypi_0 pypi pytest 5.4.3 pypi_0 pypi pytest-benchmark 3.2.3 pypi_0 pypi python 3.7.16 h7a1cb2a_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main python-dateutil 2.8.1 pypi_0 pypi python_abi 3.7 2_cp37m conda-forge pytz 2020.1 pypi_0 pypi pywavelets 1.1.1 pypi_0 pypi pyyaml 5.3.1 pypi_0 pypi pyzmq 19.0.1 pypi_0 pypi qtconsole 5.0.2 pypi_0 pypi qtpy 1.9.0 pypi_0 pypi readline 8.2 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main requests 2.23.0 pypi_0 pypi sacred 0.8.1 pypi_0 pypi scikit-image 0.17.2 pypi_0 pypi scipy 1.4.1 pypi_0 pypi seaborn 0.10.1 pypi_0 pypi send2trash 1.5.0 pypi_0 pypi setuptools 65.6.3 py37h06a4308_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main six 1.15.0 pypi_0 pypi smmap 3.0.4 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main submitit 1.1.5 pypi_0 pypi terminado 0.9.2 pypi_0 pypi testpath 0.4.4 pypi_0 pypi tifffile 2020.6.3 pypi_0 pypi tk 8.6.12 h1ccaba5_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main toml 0.10.2 pypi_0 pypi torch 1.8.1+cu111 pypi_0 pypi torchaudio 0.8.1 pypi_0 pypi torchfile 0.1.0 pypi_0 pypi torchvision 0.9.1+cu111 pypi_0 pypi tornado 6.1 pypi_0 pypi tqdm 4.46.1 pypi_0 pypi traitlets 5.0.5 pypi_0 pypi typed-ast 1.4.1 pypi_0 pypi typing-extensions 3.7.4.3 pypi_0 pypi urllib3 1.25.9 pypi_0 pypi visdom 0.1.8.9 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 0.57.0 pypi_0 pypi wheel 0.38.4 py37h06a4308_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main widgetsnbextension 3.5.1 pypi_0 pypi wrapt 1.12.1 pypi_0 pypi xmltodict 0.12.0 pypi_0 pypi xz 5.4.2 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main zipp 3.1.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main
