too large launch parameter: AddTakeGrad[130560,1], [64,1,1]

[KevinQian97]

Hi, I met a strange problem when setting the scale of the image from your [1000,600] to [1920, 1080] and get the error report below:

Traceback (most recent call last): File "dff_rfcn/train_end2end.py", line 179, in main() File "dff_rfcn/train_end2end.py", line 176, in main config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step) File "dff_rfcn/train_end2end.py", line 169, in train_net arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch) File "/Deep-Feature-Flow/dff_rfcn/core/module.py", line 981, in fit self.update_metric(eval_metric, data_batch.label) File "/Deep-Feature-Flow/dff_rfcn/core/module.py", line 1073, in update_metric self._curr_module.update_metric(eval_metric, labels) File "/Deep-Feature-Flow/dff_rfcn/core/module.py", line 674, in update_metric self._exec_group.update_metric(eval_metric, labels) File "/Deep-Feature-Flow/dff_rfcn/core/DataParallelExecutorGroup.py", line 481, in update_metric eval_metric.update(labels, texec.outputs) File "/usr/local/lib/python2.7/dist-packages/mxnet/metric.py", line 318, in update metric.update(labels, preds) File "/Deep-Feature-Flow/dff_rfcn/core/metric.py", line 51, in update pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32') File "/usr/local/lib/python2.7/dist-packages/mxnet/ndarray/ndarray.py", line 1980, in asnumpy ctypes.c_size_t(data.size))) File "/usr/local/lib/python2.7/dist-packages/mxnet/base.py", line 252, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [01:12:23] /work/mxnet/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:58: too large launch parameter: AddTakeGrad[130560,1], [64,1,1]

Stack trace returned 10 entries: [bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x4114ba) [0x7f89e3d464ba] [bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x411ad1) [0x7f89e3d46ad1] [bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x4b7eddb) [0x7f89e84b3ddb] [bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x4e12eec) [0x7f89e8747eec] [bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x4e132cf) [0x7f89e87482cf] [bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x4f92da4) [0x7f89e88c7da4] [bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2ceb179) [0x7f89e6620179] [bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2cf1e67) [0x7f89e6626e67] [bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2cd01c4) [0x7f89e66051c4] [bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2cd44b3) [0x7f89e66094b3]

I tried to locate the problem and found it happened in /Deep-Feature-Flow/dff_rfcn/core/metric.py line46 pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32') if you delete asnumpy() then the problem get solved but the whole program will run much slower and many other codes need to be changed. Would you mind telling me why this error happened and if there are other methods to fix the problem?

Thanks for your help!