SSD-Tensorflow
SSD-Tensorflow copied to clipboard
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[16,64,300,300] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2018-07-28 10:35:04.192687: W c:\users\user\source\repos\tensorflow\tensorflow\core\framework\op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:673 : Resource exhausted: OOM when allocating tensor with shape[16,64,300,300] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[16,64,300,300] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: ssd_300_vgg/conv1/conv1_1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fifo_queue_Dequeue/_497, ssd_300_vgg/conv1/conv1_1/weights/read/_495)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: gradients/ssd_300_vgg/block10/conv1x1/BiasAdd_grad/BiasAddGrad/_749 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1185_gradients/ssd_300_vgg/block10/conv1x1/BiasAdd_grad/BiasAddGrad", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'ssd_300_vgg/conv1/conv1_1/Conv2D', defined at:
File "D:/SSD-Tensorflow-master-2/train_ssd_network.py", line 411, in
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[16,64,300,300] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: ssd_300_vgg/conv1/conv1_1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fifo_queue_Dequeue/_497, ssd_300_vgg/conv1/conv1_1/weights/read/_495)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: gradients/ssd_300_vgg/block10/conv1x1/BiasAdd_grad/BiasAddGrad/_749 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1185_gradients/ssd_300_vgg/block10/conv1x1/BiasAdd_grad/BiasAddGrad", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
how to solve this problem?
Oh,I am also faced with this problem,can anyone help me?
I have solved this problem. We just need a smaller batch_size,such as 16.
your memory of GPU is not enough, you can change a smaller batch_size
I have solved this problem. We just need a smaller batch_size,such as 16.
SENİ COK SEVIYORM.. THAT MEANS I LOVE YOU
I am getting this error while training with batch size = 1 on kaggle. Anyone please help!!!
I am getting this error while training with batch size = 1 on kaggle. Anyone please help!!!
Me too~ Hope anyone help us
I am getting this error while training with batch size = 1 on kaggle. Anyone please help!!!
Me too~ Hope anyone help us
I solved it by reducing the complexity of the Ann architecture a little bit.