FaceRank icon indicating copy to clipboard operation
FaceRank copied to clipboard

model放好了 run_model 出现这个错误 把run_mode.pyl里的batch_size值改小 还是报这个错

Open lvZic opened this issue 7 years ago • 1 comments

2018-03-13 15:20:04.122000: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3 6\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats: Limit: 1631711232 InUse: 1229069056 MaxInUse: 1631711232 NumAllocs: 51 MaxAllocSize: 422898176

2018-03-13 15:20:04.122000: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3 6\tensorflow\core\common_runtime\bfc_allocator.cc:279] _______________________ **************************************************************************x 2018-03-13 15:20:04.123000: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3 6\tensorflow\core\framework\op_kernel.cc:1202] OP_REQUIRES failed at assign_op.h :111 : Resource exhausted: OOM when allocating tensor with shape[98304,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_b fc Traceback (most recent call last): File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie nt\session.py", line 1361, in _do_call return fn(*args) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie nt\session.py", line 1340, in _run_fn target_list, status, run_metadata) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram ework\errors_impl.py", line 516, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat ing tensor with shape[98304,1024] and type float on /job:localhost/replica:0/tas k:0/device:GPU:0 by allocator GPU_0_bfc [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/ device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "run_model.py", line 117, in saver.restore(sess, "./model/model.ckpt") File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 1755, in restore {self.saver_def.filename_tensor_name: save_path}) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie nt\session.py", line 905, in run run_metadata_ptr) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie nt\session.py", line 1137, in _run feed_dict_tensor, options, run_metadata) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie nt\session.py", line 1355, in _do_run options, run_metadata) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie nt\session.py", line 1374, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat ing tensor with shape[98304,1024] and type float on /job:localhost/replica:0/tas k:0/device:GPU:0 by allocator GPU_0_bfc [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/ device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'save/Assign_7', defined at: File "run_model.py", line 113, in saver=tf.train.Saver() File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 1293, in init self.build() File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 1302, in build self._build(self._filename, build_save=True, build_restore=True) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 1339, in _build build_save=build_save, build_restore=build_restore) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 796, in _build_internal restore_sequentially, reshape) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 471, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai ning\saver.py", line 161, in restore self.op.get_shape().is_fully_defined()) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\ops
state_ops.py", line 280, in assign validate_shape=validate_shape) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\ops
gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram ework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram ework\ops.py", line 3271, in create_op op_def=op_def) File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram ework\ops.py", line 1650, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected- access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor wit h shape[98304,1024] and type float on /job:localhost/replica:0/task:0/device:GPU :0 by allocator GPU_0_bfc [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/ device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

lvZic avatar Mar 13 '18 07:03 lvZic

是因为GPU大小不够用了吗

lvZic avatar Mar 13 '18 07:03 lvZic