An error training an Encoder-Decoder Attention Model

[qiang2100]

When I train an Encoder-Decoder Attention Model using "sh run_std.sh", I get the following error:

/home/qiang/torch/extra/cutorch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [56,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed. THCudaCheck FAIL file=/home/qiang/torch/extra/cutorch/lib/THC/generic/THCStorage.c line=32 error=59 : device-side assert triggered /home/qiang/torch/install/bin/luajit: cuda runtime error (59) : device-side assert triggered at /home/qiang/torch/extra/cutorch/lib/THC/generic/THCStorage.c:32 stack traceback: [C]: at 0x7fbc8f5b6050 [C]: in function '__index' layers/EMaskedClassNLLCriterion.lua:18: in function 'forward' nnets/EncDecAWE.lua:391: in function 'opfunc' /home/qiang/torch/install/share/lua/5.1/optim/sgd.lua:44: in function 'optimMethod' nnets/EncDecAWE.lua:468: in function 'trainBatch' train.lua:40: in function 'train' train.lua:162: in function 'main' train.lua:269: in function 'main' train.lua:272: in main chunk [C]: in function 'dofile' ...iang/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk [C]: at 0x00405e90 Lock freed

Usage instructions:

To obtain and lock an id: ./gpu_lock.py --id The lock is automatically freed when the parent terminates

To get an id that won't be freed: ./gpu_lock.py --id-to-hog You must manually free these ids: ./gpu_lock.py --free

More info: http://homepages.inf.ed.ac.uk/imurray2/code/gpu_monitoring/

[Sanqiang]

If you change to CPU mode and you can see more clearly the error comes from. One of bug I fixed is maybe because the author uses an older version of Torch. I fix my bug by replacing float to double.

[Crista23]

Hi @qiang2100 ! I am encountering the same error, did you find what is causing it?