iperov
Out of memory issue while gpu is only at 5 gb vram usage
Hi everyone, when I try to use SAEHD with my 1080 ti (11GB) with windows 10 then it gives me the following error:
Running trainer.
[new] No saved models found. Enter a name of a new model : new
Model first run.
Choose one or several GPU idxs (separated by comma).
[CPU] : CPU [0] : NVIDIA GeForce GTX 1080 Ti
[0] Which GPU indexes to choose? : 0
[0] Autobackup every N hour ( 0..24 ?:help ) : 0 [n] Write preview history ( y/n ?:help ) : n [120000] Target iteration : 120000 [n] Flip SRC faces randomly ( y/n ?:help ) : n [y] Flip DST faces randomly ( y/n ?:help ) : y [6] Batch_size ( ?:help ) : 8 8 [320] Resolution ( 64-640 ?:help ) : 320 [f] Face type ( h/mf/f/wf/head ?:help ) : f [df-ud] AE architecture ( ?:help ) : df-ud [320] AutoEncoder dimensions ( 32-1024 ?:help ) : 320 [72] Encoder dimensions ( 16-256 ?:help ) : 72 [72] Decoder dimensions ( 16-256 ?:help ) : 72 [16] Decoder mask dimensions ( 16-256 ?:help ) : 16 [n] Eyes and mouth priority ( y/n ?:help ) : n [n] Uniform yaw distribution of samples ( y/n ?:help ) : n [n] Blur out mask ( y/n ?:help ) : n [y] Place models and optimizer on GPU ( y/n ?:help ) : n [n] Use AdaBelief optimizer? ( y/n ?:help ) : n [n] Use learning rate dropout ( n/y/cpu ?:help ) : n [y] Enable random warp of samples ( y/n ?:help ) : y [0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) : 0.0 [0.0] GAN power ( 0.0 .. 5.0 ?:help ) : 0.0 [0.01] 'True face' power. ( 0.0000 .. 1.0 ?:help ) : 0.01 [0.0] Face style power ( 0.0..100.0 ?:help ) : 0.0 [0.0] Background style power ( 0.0..100.0 ?:help ) : 0.0 [lct] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) : none none [n] Enable gradient clipping ( y/n ?:help ) : n [n] Enable pretraining mode ( y/n ?:help ) : n Initializing models: 100%|###############################################################| 7/7 [00:04<00:00, 1.43it/s] Loading samples: 100%|###############################################################| 349/349 [00:05<00:00, 59.79it/s] Loading samples: 100%|#############################################################| 1446/1446 [00:19<00:00, 74.43it/s] ==================== Model Summary ==================== == == == Model name: new_SAEHD == == == == Current iteration: 0 == == == ==------------------ Model Options ------------------== == == == resolution: 320 == == face_type: f == == models_opt_on_gpu: False == == archi: df-ud == == ae_dims: 320 == == e_dims: 72 == == d_dims: 72 == == d_mask_dims: 16 == == masked_training: True == == eyes_mouth_prio: False == == uniform_yaw: False == == blur_out_mask: False == == adabelief: False == == lr_dropout: n == == random_warp: True == == random_hsv_power: 0.0 == == true_face_power: 0.01 == == face_style_power: 0.0 == == bg_style_power: 0.0 == == ct_mode: none == == clipgrad: False == == pretrain: False == == autobackup_hour: 0 == == write_preview_history: False == == target_iter: 120000 == == random_src_flip: False == == random_dst_flip: True == == batch_size: 8 == == gan_power: 0.0 == == gan_patch_size: 40 == == gan_dims: 16 == == == ==------------------- Running On --------------------== == == == Device index: 0 == == Name: NVIDIA GeForce GTX 1080 Ti == == VRAM: 9.41GB == == ==
Starting. Target iteration: 120000. Press "Enter" to stop training and save model.
Trying to do the first iteration. If an error occurs, reduce the model parameters.
!!! Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly. https://i.imgur.com/B7cmDCB.jpg !!! You are training the model from scratch. It is strongly recommended to use a pretrained model to speed up the training and improve the quality.
Error: OOM when allocating tensor with shape[8,576,80,80] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Add_45 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Conv2D.py:107) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node gradients_1/AddN_39 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\ops\__init__.py:55) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'Add_45', defined at: File "threading.py", line 884, in _bootstrap File "threading.py", line 916, in _bootstrap_inner File "threading.py", line 864, in run File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 410, in on_initialize gpu_pred_dst_dst, gpu_pred_dst_dstm = self.decoder_dst(gpu_dst_code) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in call return self.forward(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 225, in forward x = self.upscale2(x) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in call return self.forward(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 71, in forward x = self.conv1(x) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\LayerBase.py", line 14, in call return self.forward(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Conv2D.py", line 107, in forward x = tf.add(x, bias) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 365, in add "Add", x=x, y=y, name=name) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func return func(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op op_def=op_def) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[8,576,80,80] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Add_45 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Conv2D.py:107) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node gradients_1/AddN_39 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\ops\__init__.py:55) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Traceback (most recent call last): File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call return fn(*args) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8,576,80,80] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Add_45}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node gradients_1/AddN_39}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 129, in trainerThread iter, iter_time = model.train_one_iter() File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 474, in train_one_iter losses = self.onTrainOneIter() File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 774, in onTrainOneIter src_loss, dst_loss = self.src_dst_train (warped_src, target_src, target_srcm, target_srcm_em, warped_dst, target_dst, target_dstm, target_dstm_em) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 584, in src_dst_train self.target_dstm_em:target_dstm_em, File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 929, in run run_metadata_ptr) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run run_metadata) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8,576,80,80] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Add_45 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Conv2D.py:107) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node gradients_1/AddN_39 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\ops\__init__.py:55) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'Add_45', defined at: File "threading.py", line 884, in _bootstrap File "threading.py", line 916, in _bootstrap_inner File "threading.py", line 864, in run File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 410, in on_initialize gpu_pred_dst_dst, gpu_pred_dst_dstm = self.decoder_dst(gpu_dst_code) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in call return self.forward(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 225, in forward x = self.upscale2(x) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in call return self.forward(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 71, in forward x = self.conv1(x) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\LayerBase.py", line 14, in call return self.forward(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Conv2D.py", line 107, in forward x = tf.add(x, bias) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 365, in add "Add", x=x, y=y, name=name) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func return func(*args, **kwargs) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op op_def=op_def) File "C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[8,576,80,80] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Add_45 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Conv2D.py:107) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node gradients_1/AddN_39 (defined at C:\Random\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\ops\__init__.py:55) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
How do I fix this? I've lowering the batch file and all but nothing works. Should I also isntall the workstation drivers instead of game drivers?
Issue solved / already answered (or it seems like user error), please close it.