Ran out of memory trying to allocate 117.19MiB. | Resource exhausted: OOM when allocating tensor.

[shrikant10]

On running this command: ./experiments/scripts/test.sh mini-vg -1
dual_graph_vrd_final 2
checkpoints/dual_graph_vrd_final_iter2.ckpt
viz_cls
0

I was getting Error: . . I tensorflow/core/common_runtime/bfc_allocator.cc:696] Sum Total of in-use chunks: 1.42GiB I tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats: Limit: 1616576512 InUse: 1519864576 MaxInUse: 1519864576 NumAllocs: 148 MaxAllocSize: 467140864

W tensorflow/core/common_runtime/bfc_allocator.cc:270] ************____***************************xxxxx W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 117.19MiB. See logs for memory state. W tensorflow/core/framework/op_kernel.cc:968] Resource exhausted: OOM when allocating tensor with shape[1,64,600,800] Traceback (most recent call last): File "./tools/test_net.py", line 91, in viz_net(args.network_name, args.model, imdb, args.test_mode) File "/home/neeraj/scene-graph-TF-release/tools/../lib/fast_rcnn/visualize.py", line 157, in viz_net bbox_reg, [inference_iter]) File "/home/neeraj/scene-graph-TF-release/tools/../lib/fast_rcnn/test.py", line 142, in im_detect ops_value = sess.run(ops, feed_dict=feed_dict) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 717, in run run_metadata_ptr) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 915, in _run feed_dict_string, options, run_metadata) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _do_run target_list, options, run_metadata) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 985, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[1,64,600,800] [[Node: conv1_2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](conv1_1/conv1_1, conv1_2/weights/read)]] [[Node: Cast_3/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_78_Cast_3", tensor_type=DT_DOUBLE, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Caused by op u'conv1_2/Conv2D', defined at: File "./tools/test_net.py", line 91, in viz_net(args.network_name, args.model, imdb, args.test_mode) File "/home/neeraj/scene-graph-TF-release/tools/../lib/fast_rcnn/visualize.py", line 111, in viz_net net.setup() File "/home/neeraj/scene-graph-TF-release/tools/../lib/networks/models.py", line 178, in setup self._vgg_conv() File "/home/neeraj/scene-graph-TF-release/tools/../lib/networks/models.py", line 37, in _vgg_conv .conv(3, 3, 64, 1, 1, name='conv1_2') File "/home/neeraj/scene-graph-TF-release/tools/../lib/networks/network.py", line 25, in layer_decorated layer_output = op(self, layer_input, *args, **kwargs) File "/home/neeraj/scene-graph-TF-release/tools/../lib/networks/network.py", line 135, in conv conv = convolve(input, kernel) File "/home/neeraj/scene-graph-TF-release/tools/../lib/networks/network.py", line 126, in convolve = lambda i, k: tf.nn.conv2d(i, k, [1, s_h, s_w, 1], padding=padding) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d data_format=data_format, name=name) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op op_def=op_def) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2380, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/neeraj/anaconda3/envs/scene-graph/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1298, in init self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,64,600,800] [[Node: conv1_2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](conv1_1/conv1_1, conv1_2/weights/read)]] [[Node: Cast_3/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_78_Cast_3", tensor_type=DT_DOUBLE, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Please Help !! @danfeiX @FrancoisPl