warp-ctc icon indicating copy to clipboard operation
warp-ctc copied to clipboard

Warp-CTC error on GPU : "cuda memcpy or memset failed"

Open kites520 opened this issue 7 years ago • 6 comments

I compile mxnet with warp-ctc plugin. My env is: Ubuntu 14.04 + CUDA 8.0 + cuDNN 5.1 + Torch 7.0, GTX960.

When I compile warp-ctc, everything is normal, and passed "warp-ctc/build/test_gpu".

I rebuild mxnet successfully, except it shows warning : "nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning)."

Whichever I execute "example/image-classification/train_mnist.py" on mx.context.cpu(0) or mx.context.gpu(0), it works normal.

But when I execute "example/warpctc/toy_ctc.py" on mx.context.gpu(0), an error occurred :

terminate called after throwing an instance of 'std::runtime_error'
  what():  Error: compute_ctc_loss, stat = cuda memcpy or memset failed"

And if I use cpu, it is OK.

How to solve this problem?THX.

kites520 avatar Nov 14 '16 06:11 kites520

same here

wanghaisheng avatar Nov 28 '16 09:11 wanghaisheng

same problem here with Ubuntu 14.04 + CUDA 8.5 + GTX TITAN (Pascal)

liueryun avatar Dec 02 '16 16:12 liueryun

same to me back to 5bfb46e83bcb338a09fbf875cb0b02bc39746b36.i am work fine @liueryun @wanghaisheng

yajiedesign avatar Jan 04 '17 15:01 yajiedesign

The API on the master branch changed and mxnet looks to be using the old API, to maintain compatibility you can use the v1 tag/branch which will maintain the old API.

The structure passed in to the ctc functions changed from ctcComputeInfo to ctcOptions. Not sure how the mxnet plugin compiled since it uses the old name. Seems like at some point mxnet was compiled against the old source code but was being linked with a warpctc binary built from the new source.

jaredcasper avatar Jan 04 '17 20:01 jaredcasper

i am pr https://github.com/dmlc/mxnet/pull/4530 to fix this. wait merge.

yajiedesign avatar Jan 05 '17 02:01 yajiedesign

same here

Jacksonlark avatar Jul 09 '17 05:07 Jacksonlark