在通过了testexample后,执行test_cuda时出现了如下问题。我尝试了多个版本的cuda和cudnn,但是仍然会发生如下同样的问题
Describe the bug
在通过了testexample后,执行test_cuda时出现了如下问题。我尝试了多个版本的cuda和cudnn,但是仍然会发生如下同样的问题。在论坛中一个类似的问题下,解决方案是升级到最新的cuda和cudnn,但我发现这对我而言并不起效。
Full Log
[i 0621 08:43:05.209275 72 lock.py:85] Create lock file:/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/jittor.lock
[i 0621 08:43:05.221116 72 compiler.py:955] Jittor(1.3.8.2) src: /home/yhr/.local/lib/python3.10/site-packages/jittor
[i 0621 08:43:05.223744 72 compiler.py:956] g++ at /usr/bin/g++(9.5.0)
[i 0621 08:43:05.223857 72 compiler.py:957] cache_path: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default
[i 0621 08:43:05.227127 72 init.py:411] Found nvcc(12.1.66) at /usr/local/cuda/bin/nvcc.
[i 0621 08:43:05.229616 72 init.py:411] Found addr2line(2.38) at /usr/bin/addr2line.
[i 0621 08:43:05.417394 72 compiler.py:1010] cuda key:cu12.1.66_sm_61
[i 0621 08:43:05.449123 72 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61
[i 0621 08:43:05.449343 72 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/jit
[i 0621 08:43:05.449489 72 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/obj_files
[i 0621 08:43:05.449634 72 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/gen
[i 0621 08:43:05.449773 72 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/tmp
[i 0621 08:43:05.449910 72 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/checkpoints
[i 0621 08:43:28.463532 24 init.py:227] Total mem: 125.61GB, using 16 procs for compiling.
Compiling jittor_core(151/151) used: 30.897s eta: 0.000s
[i 0621 08:43:59.815253 24 jit_compiler.cc:28] Load cc_path: /usr/bin/g++
[i 0621 08:44:00.097034 24 init.cc:62] Found cuda archs: [61,]
[i 0621 08:44:01.531990 24 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/cutt
[i 0621 08:44:01.596554 24 compile_extern.py:339] Downloading cutt...
Downloading https://codeload.github.com/Jittor/cutt/zip/v1.2 to /home/yhr/.cache/jittor/cutt/cutt-1.2.zip
296kB [00:00, 324kB/s]
[i 0621 08:44:02.551298 24 compile_extern.py:352] installing cutt...
Compiling libcutt(9/9) used: 10.570s eta: 0.000s
[i 0621 08:44:13.259755 24 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/custom_ops
[i 0621 08:44:14.928621 24 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/mkl
[i 0621 08:44:14.928966 24 compile_extern.py:73] Downloading mkl...
Downloading https://cg.cs.tsinghua.edu.cn/jittor/assets/dnnl_lnx_2.2.0_cpu_gomp.tgz to /home/yhr/.cache/jittor/mkl/dnnl_lnx_2.2.0_cpu_gomp.tgz
10.5MB [00:01, 9.49MB/s]
Use time: 118.11 ms per iteration.
Example passed on CPU.
Compiling gen_ops_mkl_conv_mkl_matmul_mkl_conv_backward_w_mk___hash484cd2(6/7) uCompiling gen_ops_mkl_conv_mkl_matmul_mkl_conv_backward_w_mk___hash484cd2(7/7) used: 2.985s eta: 0.000s
[i 0621 08:44:34.660270 24 compiler.py:34] Create cache dir: /home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/cuda
Compiling libcuda_extern(3/3) used: 2.214s eta: 0.000s
Compiling gen_ops_cub_argsort_cub_where_cub_arg_reduce_cub_c___hash618137(6/6) used: 2.563s eta: 0.000s
Compiling gen_ops_cublas_acc_matmul_cublas_test_cublas_batch___hash77942d(8/8) used: 2.287s eta: 0.000s
Compiling gen_ops_cudnn_rnn_backward_x_cudnn_conv_backward_x___hashcf03c5(16/16) used: 5.119s eta: 0.000s
[i 0621 08:44:52.055809 24 cuda_flags.cc:39] CUDA enabled.
Compiling Operators(1/1) used: 3.77s eta: 0s .[i 0621 08:44:57.943756 24 cuda_flags.cc:39] CUDA enabled.
Compiling Operators(1/1) used: 4.39s eta: 0s .[i 0621 08:45:02.339329 24 cuda_flags.cc:39] CUDA enabled. /home/yhr/.local/lib/python3.10/site-packages/jittor/src/misc/cuda_atomic.h(138): error: no instance of overloaded function "atomicCAS" matches the argument list argument types are: (unsigned short *, unsigned short, unsigned short) old = atomicCAS(a_i, assume, int_mapper<__half>::to_int(b)); ^
/home/yhr/.local/lib/python3.10/site-packages/jittor/src/misc/cuda_atomic.h(153): error: no instance of overloaded function "atomicCAS" matches the argument list argument types are: (unsigned short *, unsigned short, unsigned short) old = atomicCAS(a_i, assume, int_mapper<__half>::to_int(b)); ^
2 errors detected in the compilation of "/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/jit/__opkey0_array__T_int32__o_2__opkey1_binary__Tx_int32__Ty_int32__Tz_int32__OP_add__opkey2____hash_861eebc2e1f0c5c2_op.cc". E[i 0621 08:45:06.008524 24 cuda_flags.cc:39] CUDA enabled. Compiling gen_ops_no_cuda(2/2) used: 2.402s eta: 0.000s .s
ERROR: test_cuda_fused_op (main.TestCuda)
Traceback (most recent call last): File "/home/yhr/.local/lib/python3.10/site-packages/jittor/test/test_cuda.py", line 108, in test_cuda_fused_op ((a+a)*2).data RuntimeError: Wrong inputs arguments, Please refer to examples(help(jt.data)).
Types of your inputs are: self = Var,
The function declarations are: inline DataView data()
Failed reason:[f 0621 08:45:06.007434 24 parallel_compiler.cc:330] Error happend during compilation: [Error] source file location:/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/jit/__opkey0_array__T_int32__o_2__opkey1_binary__Tx_int32__Ty_int32__Tz_int32__OP_add__opkey2____hash_861eebc2e1f0c5c2_op.cc Compile fused operator(0/1)failed:[Op(18:0:1:1:i0:o1:s0,array->19),Op(16:0:1:1:i2:o1:s0,binary.add->17),Op(22:0:1:1:i1:o1:s0,broadcast_to->23),Op(24:0:1:1:i2:o1:s0,binary.multiply->25),]
Reason: [f 0621 08:45:06.007049 24 log.cc:608] Check failed ret(256) == 0(0) Run cmd failed: "/usr/local/cuda/bin/nvcc" "/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/jit/__opkey0_array__T_int32__o_2__opkey1_binary__Tx_int32__Ty_int32__Tz_int32__OP_add__opkey2____hash_861eebc2e1f0c5c2_op.cc" -std=c++14 -Xcompiler -fPIC -Xcompiler -march=native -Xcompiler -fdiagnostics-color=always -lstdc++ -ldl -shared -I"/home/yhr/.local/lib/python3.10/site-packages/jittor/src" -I/usr/include/python3.10 -I/usr/include/python3.10 -DHAS_CUDA -DIS_CUDA -I"/usr/local/cuda/include" -I"/home/yhr/.local/lib/python3.10/site-packages/jittor/extern/cuda/inc" -lcudart -L"/usr/local/cuda/lib64" -Xlinker -rpath="/usr/local/cuda/lib64" -I"/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61" -L"/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61" -Xlinker -rpath="/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61" -L"/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default" -Xlinker -rpath="/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default" -l:"jit_utils_core.cpython-310-x86_64-linux-gnu".so -l:"jittor_core.cpython-310-x86_64-linux-gnu".so -x cu --cudart=shared -ccbin="/usr/bin/g++" --use_fast_math -w -I"/home/yhr/.local/lib/python3.10/site-packages/jittor/extern/cuda/inc" -arch=compute_61 -code=sm_61 -o "/home/yhr/.cache/jittor/jt1.3.8/g++9.5.0/py3.10.6/Linux-5.15.0-7x2f/IntelRXeonRCPUx6c/default/cu12.1.66_sm_61/jit/__opkey0_array__T_int32__o_2__opkey1_binary__Tx_int32__Ty_int32__Tz_int32__OP_add__opkey2____hash_861eebc2e1f0c5c2_op.so"
Minimal Reproduce
我使用了版本如下
Ubuntu 22.02
python 3.10
g++ 9.5
Cuda 11.3 + cudnn 8.8.0
or Cuda 12.1 + cudnn 8.9.1
Expected behavior
请求各位帮助我发现任何可能存在的问题,并告诉我可能解决问题的办法
你解决了吗,我也遇到了“"atomicCAS"”这个报错。
我也出现了同样的问题,难道是ubuntu版本的原因
w
我也出现了同样的问题,难道是ubuntu版本的原因
@ginogogo @Y-huange @yhr20000319 请问你们解决了吗?我也出现了同样的问题
w
我也出现了同样的问题,难道是ubuntu版本的原因
@ginogogo @Y-huange @yhr20000319 请问你们解决了吗?我也出现了同样的问题
已解决,思路来自https://discuss.jittor.org/t/topic/475 将jittor版本下降到1.3.6.5即可,我的版本Ubuntu22.04 Cuda11.7 python3.8
已在最新的代码里修复
这是来自QQ邮箱的假期自动回复邮件。 您好,我最近正在休假中,无法亲自回复您的邮件。我将在假期结束后,尽快给您回复。