practical_seq2seq
practical_seq2seq copied to clipboard
Published
20 hours ago •
suriyadeepan
suriyadeepan
OOM when allocating tensor with shape[2048,4096]
When i try to run python 04-Cornell-Movie-Dialog-Bot.py get this error:
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000079CFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000079EFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A0FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A2FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A4FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A6FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007A8FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007AAFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007ACFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007AEFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B0FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B2FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B4FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B6FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007B8FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007BAFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007BCFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007BEFE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007C0FE2100 of size 33554432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000007C2FE2100 of size 36160768
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710DC3600 of size 8448
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710DC5800 of size 5120
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710DC6D00 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710E03700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710EE3800 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710F23900 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000710F63C00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000711143D00 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007111A3D00 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007112C3D00 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F51C600 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F6FC600 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F798600 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F7BC600 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F8BC600 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073F91C600 of size 917504
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FA1C600 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FB40600 of size 901120
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FC3C600 of size 3252480
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000073FF96700 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740076700 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007400B6700 of size 131072
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740196700 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074023A700 of size 3236096
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740570800 of size 786432
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740634800 of size 1818624
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007407F4800 of size 2449664
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740A6A900 of size 1179648
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000740BAA900 of size 5480704
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000741104A00 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007411C4A00 of size 3907840
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074159EB00 of size 655360
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074165EB00 of size 393216
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007416DEB00 of size 393216
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000074175EB00 of size 3121408
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000741A78C00 of size 5480704
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 0000000741FF2D00 of size 1835008
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000007421D2D00 of size 4038912
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:693] Summary of in-use Chunks by size:
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 25 Chunks of size 256 totalling 6.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 6400 totalling 6.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 93 Chunks of size 16384 totalling 1.45MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 5 Chunks of size 32256 totalling 157.5KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 678 Chunks of size 131072 totalling 84.75MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 241152 totalling 235.5KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 257792 totalling 251.8KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 51 Chunks of size 262144 totalling 12.75MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 35 Chunks of size 393216 totalling 13.13MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 638976 totalling 624.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 7 Chunks of size 32776192 totalling 218.80MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 81 Chunks of size 33554432 totalling 2.53GiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 35646464 totalling 34.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 36160768 totalling 34.49MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 42034944 totalling 40.09MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 66330624 totalling 63.26MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:700] Sum Total of in-use chunks: 3.02GiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:702] Stats:
Limit: 3292128870
InUse: 3246350848
MaxInUse: 3249725952
NumAllocs: 7135
MaxAllocSize: 66330624
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:274] **************************************************************************
**************************
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:275] Ran out of memory trying to allocate 32.00MiB. See logs for memory state.
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[2048,4096]
Traceback (most recent call last):
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _do_call
return fn(*args)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1003, in _run_fn
status, run_metadata)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 469, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,4096]
[[Node: gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=true,
transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/concat, gradients
/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/add_grad/tuple/control_dependency)]]
[[Node: Adam/update_decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam/_410 = _Recv[client_terminated=false, recv_d
evice="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_656_Adam/update_decoder/embedding_rnn_seq2seq/e
mbedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "04-Cornell-Movie-Dialog-Bot.py", line 42, in <module>
sess = model.train(train_batch_gen, val_batch_gen)
File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 144, in train
self.train_batch(sess, train_set)
File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 102, in train_batch
_, loss_v = sess.run([self.train_op, self.loss], feed_dict)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 766, in run
run_metadata_ptr)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,4096]
[[Node: gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=true,
transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/concat, gradients
/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/add_grad/tuple/control_dependency)]]
[[Node: Adam/update_decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam/_410 = _Recv[client_terminated=false, recv_d
evice="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_656_Adam/update_decoder/embedding_rnn_seq2seq/e
mbedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op 'gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1', defined at:
File "04-Cornell-Movie-Dialog-Bot.py", line 30, in <module>
num_layers=3
File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 79, in __init__
__graph__()
File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 75, in __graph__
self.train_op = tf.train.AdamOptimizer(learning_rate=lr).minimize(self.loss)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\optimizer.py", line 269, in minimize
grad_loss=grad_loss)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\optimizer.py", line 335, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 482, in gradients
in_grads = grad_fn(op, *out_grads)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_grad.py", line 731, in _MatMulGrad
math_ops.matmul(op.inputs[0], grad, transpose_a=True))
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1729, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1442, in _mat_mul
transpose_b=transpose_b, name=name)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 759, in apply_op
op_def=op_def)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1128, in __init__
self._traceback = _extract_stack()
...which was originally created as op 'decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul', defined at:
File "04-Cornell-Movie-Dialog-Bot.py", line 30, in <module>
num_layers=3
[elided 0 identical lines from previous traceback]
File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 79, in __init__
__graph__()
File "c:\Repositories\practical_seq2seq\seq2seq_wrapper.py", line 58, in __graph__
xvocab_size, yvocab_size, emb_dim)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\seq2seq.py", line 357, in embedding_rnn_seq2seq
feed_previous=feed_previous)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\seq2seq.py", line 280, in embedding_rnn_decoder
loop_function=loop_function)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\seq2seq.py", line 146, in rnn_decoder
output, state = cell(inp, state)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 586, in __call__
output, res_state = self._cell(inputs, state)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 815, in __call__
cur_inp, new_state = cell(cur_inp, cur_state)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 684, in __call__
output, new_state = self._cell(inputs, state, scope)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 311, in __call__
concat = _linear([inputs, h], 4 * self._num_units, True)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\rnn_cell.py", line 909, in _linear
res = math_ops.matmul(array_ops.concat(1, args), matrix)
File "C:\Users\felansu\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1729, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2048,4096]
[[Node: gradients/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, transpose_a=true,
transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/Linear/concat, gradients
/decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell_4/Cell0/BasicLSTMCell/add_grad/tuple/control_dependency)]]
[[Node: Adam/update_decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam/_410 = _Recv[client_terminated=false, recv_d
evice="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_656_Adam/update_decoder/embedding_rnn_seq2seq/e
mbedding_rnn_decoder/rnn_decoder/MultiRNNCell/Cell0/BasicLSTMCell/Linear/Bias/ApplyAdam", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_gpu_executor.cc:652] Deallocating stream with pending work
I keep getting the same error too. Running this on Google cloud instance with 2 GPU on a train dataset with only 40K data. Was wondering if anyone found the issue or solution for this problem.
