math_functions.cu:130] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR

[takecareofbigboss]

math_functions.cu:130] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR *** Check failure stack trace: *** @ 0x7f2a5c719b4d google::LogMessage::Fail() @ 0x7f2a5c71db67 google::LogMessage::SendToLog() @ 0x7f2a5c71b9e9 google::LogMessage::Flush() @ 0x7f2a5c71bced google::LogMessageFatal::~LogMessageFatal() @ 0x7f2a63d10256 caffe::caffe_gpu_dot<>() @ 0x7f2a63cd4ffc caffe::SoftmaxWithLossLayer<>::Forward_gpu() @ 0x7f2a63b108cc caffe::Net<>::ForwardFromTo() @ 0x7f2a63b10ca7 caffe::Net<>::Forward() @ 0x7f2a6398a308 caffe::Solver<>::Step() @ 0x7f2a63ae78ea caffe::Worker<>::InternalThreadEntry() @ 0x7f2a63afb0d0 caffe::InternalThread::entry() @ 0x7f2a63afb8f6 boost::detail::thread_data<>::run() @ 0x7f2a5800ee83 thread_proxy @ 0x7f2a4abbb1c3 start_thread @ 0x7f2a4a8ed12d __clone

when i use single gpu, the model can train normaly. But when I use multi-gpu to train "sh examples/FRCNN/zf/train_frcnn.sh", it reminds me that ...

[takecareofbigboss]

it occurs when i set this: gpu=all !!!!!!!!!!!!!!!!!!!!!!!!!!!! so, your code can not train with multi gpu?

if [ ! -n "$1" ] ;then echo "$1 is empty, default is 0" gpu=all else echo "use $1-th gpu" gpu=$1 fi