TurboTransformers icon indicating copy to clipboard operation
TurboTransformers copied to clipboard
verified publisher icon Tencent

cuda error in Transformer decoder in TurboTransformer V0.3.0

Open aihebaikaishui opened this issue 5 years ago • 3 comments

when we use the Transformer decoder in TurboTransformer V0.3.0, a cuda error appears. The error is shown below.

RuntimeError: CUDA error: an illegal memory access was encountered [TT_ERROR] CUDA runtime error: an illegal memory access was encountered /TurboTransformers/turbo_transformers/core/cuda_device_context.cpp:33

We want to apply the TurboTransformer V0.3.0 in Machine Translation, but can not solve this issue. Thanks for any suggestions!

aihebaikaishui avatar Oct 12 '20 03:10 aihebaikaishui

Hi, could you update your version to the lastest version v0.4.2? If you still need v0.3.0 support, can you give a commit hash id? I am willing to help you identify the malloc bug in that commit.

feifeibear avatar Oct 12 '20 03:10 feifeibear

Hello, thank you for your advice. We have update the version from previous v0.3.0 to lastest version v0.4.2, but the problem still exists. We still face with the CUDA error: [TT_ERROR] CUDA runtime error: an illegal memory access was encountered /workspace/TurboTransformers/turbo_transformers/core/cuda_device_context.cpp:33. We also tried to solve this problem by rebuilding the docker and container, however, it doesn't work. I would appreciate it if you help us to solve this problem. Looking forward to your reply.

aihebaikaishui avatar Oct 19 '20 02:10 aihebaikaishui

Hi, I got it. Can you paste your build commands as well as the run scripts which may help me to reproduce the error? I guess you are working on the decoder code, which may be not well-tested.

feifeibear avatar Oct 19 '20 02:10 feifeibear