File "/home/sh/anaconda3/envs/LSKNet/lib/python3.8/site-packages/torchsort/ops.py", line 119, in backward grad -= isotonic_l2_backward[s.device.type]( RuntimeError: CUDA error: an illegal memory access was encountered
Sudden error during training environment: Package Version Editable project location
addict 2.4.0 aliyun-python-sdk-core 2.16.0 aliyun-python-sdk-kms 2.16.5 blessed 1.20.0 certifi 2024.8.30 cffi 1.17.1 charset-normalizer 3.4.0 click 8.1.7 colorama 0.4.6 contourpy 1.1.1 crcmod 1.7 cryptography 43.0.1 cycler 0.12.1 Cython 3.0.11 e2cnn 0.2.3 einops 0.8.0 filelock 3.14.0 fonttools 4.54.1 gpustat 1.1.1 idna 3.10 importlib_metadata 8.5.0 importlib_resources 6.4.5 jmespath 0.10.0 kiwisolver 1.4.7 Markdown 3.7 markdown-it-py 3.0.0 matplotlib 3.7.5 mdurl 0.1.2 mmcv-full 1.7.2 mmdet 2.28.2 mmrotate 0.3.4 /home/sh/mmrotate model-index 0.1.11 mpmath 1.3.0 numpy 1.24.4 nvidia-ml-py 12.560.30 opencv-python 4.10.0.84 opendatalab 0.0.10 openmim 0.3.9 openxlab 0.1.2 ordered-set 4.1.0 oss2 2.17.0 packaging 24.1 pandas 2.0.3 pillow 10.4.0 pip 24.2 platformdirs 4.3.6 prettytable 3.11.0 psutil 6.0.0 pycocotools 2.0.7 pycparser 2.22 pycryptodome 3.21.0 Pygments 2.18.0 pyparsing 3.1.4 python-dateutil 2.9.0.post0 pytz 2023.4 PyYAML 6.0.2 requests 2.28.2 rich 13.4.2 scipy 1.10.1 setuptools 60.2.0 shapely 2.0.6 six 1.16.0 sympy 1.13.3 tabulate 0.9.0 terminaltables 3.1.10 tomli 2.0.2 torch 1.9.1+cu111 torchaudio 0.9.1 torchsort 0.1.9 torchvision 0.10.1+cu111 tqdm 4.65.2 typing_extensions 4.12.2 tzdata 2024.2 urllib3 1.26.20 wcwidth 0.2.13 wheel 0.44.0 yapf 0.40.2 zipp 3.20.2