src/engine/./threaded_engine.h:359: [03:10:06] src/storage/./pooled_storage_manager.h:107: cudaMalloc failed: out of memory

[caoqichun]

Ubuntu16.04 Processor: i5-7500CPU @3.40GHz x 4 Memory:15.6G Graphics: GTX960 4G I am run "./run.sh". I am get this error:

INFO:root:lr 0.001000 lr_epoch_diff [12, 18] lr_iters [984960, 1477440] [03:09:44] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) [03:09:57] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) terminate called after throwing an instance of 'dmlc::Error' what(): [03:10:07] src/engine/./threaded_engine.h:359: [03:10:06] src/storage/./pooled_storage_manager.h:107: cudaMalloc failed: out of memory

Stack trace returned 10 entries: [bt] (0) /mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::StackTraceabi:cxx11+0x5a) [0x7f0720cab81a] [bt] (1) /mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f0720cac3b8] [bt] (2) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::storage::GPUPooledStorageManager::Alloc(mxnet::Storage::Handle*)+0x16d) [0x7f072370ee1d] [bt] (3) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::StorageImpl::Alloc(mxnet::Storage::Handle*)+0x5d) [0x7f07237124ed] [bt] (4) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Resource::get_space_internal(unsigned long) const+0x6e) [0x7f072381090e] [bt] (5) /mxnet/python/mxnet/../../lib/libmxnet.so(mshadow::Tensor<mshadow::gpu, 1, float> mxnet::Resource::get_space_typed<mshadow::gpu, 1, float>(mshadow::Shape<1>, mshadow::Streammshadow::gpu*) const+0xca) [0x7f07239368ea] [bt] (6) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::op::CuDNNConvolutionOp::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x20a) [0x7f072398a26a] [bt] (7) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::op::OperatorState::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x363) [0x7f07231f7da3] [bt] (8) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::StatefulComputeExecutor::Run(mxnet::RunContext, bool)+0x69) [0x7f07237b3419] [bt] (9) /mxnet/python/mxnet/../../lib/libmxnet.so(+0x33c9040) [0x7f0723786040]

A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 9 entries: [bt] (0) /mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::StackTraceabi:cxx11+0x5a) [0x7f0720cab81a] [bt] (1) /mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f0720cac3b8] [bt] (2) /mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x332) [0x7f0723702182] [bt] (3) /mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context, bool, mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>, std::shared_ptrmxnet::engine::ThreadPool::SimpleEvent)+0xcb) [0x7f072370a20b] [bt] (4) /mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptrmxnet::engine::ThreadPool::SimpleEvent), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock, bool)::{lambda()#3}::operator()() const::{lambda(std::shared_ptrmxnet::engine::ThreadPool::SimpleEvent)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptrmxnet::engine::ThreadPool::SimpleEvent&&)+0x63) [0x7f072370a403] [bt] (5) /mxnet/python/mxnet/../../lib/libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptrmxnet::engine::ThreadPool::SimpleEvent)> (std::shared_ptrmxnet::engine::ThreadPool::SimpleEvent)> >::_M_run()+0x4a) [0x7f07237045da] [bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f0704c90c80] [bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f07313676ba] [bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f073109d3dd]

./run.sh: line 2: 25842 Aborted (core dumped) python maskrcnn_train_end2end.py --gpus 0 --prefix model/e2e_pool_fullcoco_0710

someone can help me?