(0) Resource exhausted: failed to allocate memory

Open Sonicsagar opened this issue 3 years ago • 0 comments

What kind of bug is this?

======================== Model Summary ======================== == == == Model name: FF Github iperov_SAEHD == == == == Current iteration: 315611 == == == ==---------------------- Model Options ----------------------== == == == resolution: 128 == == face_type: f == == models_opt_on_gpu: True == == archi: liae == == ae_dims: 128 == == e_dims: 80 == == d_dims: 48 == == d_mask_dims: 16 == == masked_training: True == == learn_mask: True == == lr_dropout: n == == random_warp: True == == gan_power: 0.0 == == true_face_power: 0.0 == == face_style_power: 0.0 == == bg_style_power: 0.0 == == ct_mode: none == == clipgrad: False == == pretrain: False == == autobackup_hour: 0 == == write_preview_history: False == == target_iter: 0 == == random_flip: True == == batch_size: 8 == == eyes_mouth_prio: True == == uniform_yaw: False == == adabelief: False == == gan_patch_size: 16 == == gan_dims: 16 == == blur_out_mask: False == == random_hsv_power: 0.0 == == == ==----------------------- Running On ------------------------== == == == Device index: 0 == == Name: NVIDIA GeForce RTX 3070 Laptop GPU == == VRAM: 5.35GB == == ==

Starting. Press "Enter" to stop training and save model. Error: 2 root error(s) found. (0) Resource exhausted: failed to allocate memory [[node mul_49 (defined at C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py:58) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

     [[concat_3/concat/_477]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

(1) Resource exhausted: failed to allocate memory [[node mul_49 (defined at C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py:58) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

0 successful operations. 0 derived errors ignored.

Errors may have originated from an input operation. Input Source operations connected to node mul_49: src_dst_opt/acc_encoder/down1/downs_3/conv1/weight_0/read (defined at C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py:35)

Input Source operations connected to node mul_49: src_dst_opt/acc_encoder/down1/downs_3/conv1/weight_0/read (defined at C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py:35)

Original stack trace for 'mul_49': File "threading.py", line 884, in _bootstrap File "threading.py", line 916, in _bootstrap_inner File "threading.py", line 864, in run File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 564, in on_initialize src_dst_loss_gv_op = self.src_dst_opt.get_update_op (nn.average_gv_list (gpu_G_loss_gvs)) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 58, in get_update_op new_a = self.rho * a + (1. - self.rho) * tf.square(g) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1076, in _run_op return tensor_oper(a.value(), *args, **kwargs) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1400, in r_binary_op_wrapper return func(x, y, name=name) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1710, in _mul_dispatch return multiply(x, y, name=name) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper return target(*args, **kwargs) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\math_ops.py", line 530, in multiply return gen_math_ops.mul(x, y, name) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 6245, in mul "Mul", x=x, y=y, name=name) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _create_op_internal op_def=op_def) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)

Traceback (most recent call last): File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call return fn(*args) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1360, in _run_fn target_list, run_metadata) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1453, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: failed to allocate memory [[{{node mul_49}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

     [[concat_3/concat/_477]]

(1) Resource exhausted: failed to allocate memory [[{{node mul_49}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\mainscripts\Trainer.py", line 129, in trainerThread iter, iter_time = model.train_one_iter() File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\ModelBase.py", line 474, in train_one_iter losses = self.onTrainOneIter() File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 774, in onTrainOneIter src_loss, dst_loss = self.src_dst_train (warped_src, target_src, target_srcm, target_srcm_em, warped_dst, target_dst, target_dstm, target_dstm_em) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 584, in src_dst_train self.target_dstm_em:target_dstm_em, File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 968, in run run_metadata_ptr) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1191, in _run feed_dict_tensor, options, run_metadata) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1369, in _do_run run_metadata) File "C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: failed to allocate memory [[node mul_49 (defined at C:\Users\Sagar\Downloads\Programs\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py:58) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

     [[concat_3/concat/_477]]

0 successful operations. 0 derived errors ignored.

Nov 03 '22 18:11 Sonicsagar