DeepFaceLab OOM Error when use DeepFaceLab_NVIDIA_build_12_22

Hi there with the new version 12_22_2020 i got OOM errors while extract faces. Details:

GPU: GTX1650Ti
VRAM: 4G
SRAM:16G
cuda_11.1.0_456.43_win10

Error information: Choose one or several GPU idxs (separated by comma). [CPU] : CPU [0] : GeForce GTX 1650 Ti [0] Which GPU indexes to choose? : 0 [wf] Face type ( f/wf/head ?:help ) : f f [0] Max number of faces from image ( ?:help ) : 0 [512] Image size ( 256-2048 ?:help ) : 256 256 [90] Jpeg quality ( 1-100 ?:help ) : 50 50 [n] Write debug images to aligned_debug? ( y/n ) : n Extracting faces... Running on GeForce GTX 1650 Ti 0%| | 0/548 [00:00<?, ?it/s] Error while processing data: Traceback (most recent call last): File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call return fn(*args) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1360, in _run_fn target_list, run_metadata) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1453, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[64,642,362] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Pad_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Add_29/_4049]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[64,642,362] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Pad_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessorBase.py", line 71, in _subprocess_run result = self.process_data (data) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Extractor.py", line 107, in process_data rects_extractor=self.rects_extractor, File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Extractor.py", line 150, in rects_stage rects = data.rects = rects_extractor.extract (rotated_image, is_bgr=True) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\facelib\S3FDExtractor.py", line 193, in extract olist = self.model.run ([ input_image[None,...] ] ) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 167, in run return nn.tf_sess.run ( self.run_output, feed_dict=feed_dict) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 968, in run run_metadata_ptr) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1191, in _run feed_dict_tensor, options, run_metadata) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1369, in _do_run run_metadata) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[64,642,362] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Pad_1 (defined at K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\layers\Conv2D.py:97) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Add_29/_4049]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[64,642,362] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Pad_1 (defined at K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\layers\Conv2D.py:97) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations. 0 derived errors ignored.

Errors may have originated from an input operation. Input Source operations connected to node Pad_1: Relu (defined at K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\facelib\S3FDExtractor.py:93)

Input Source operations connected to node Pad_1: Relu (defined at K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\facelib\S3FDExtractor.py:93)

Original stack trace for 'Pad_1': File "", line 1, in File "multiprocessing\spawn.py", line 105, in spawn_main File "multiprocessing\spawn.py", line 118, in _main File "multiprocessing\process.py", line 258, in _bootstrap File "multiprocessing\process.py", line 93, in run File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessorBase.py", line 62, in _subprocess_run self.on_initialize(client_dict) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Extractor.py", line 73, in on_initialize self.rects_extractor = facelib.S3FDExtractor(place_model_on_cpu=place_model_on_cpu) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\facelib\S3FDExtractor.py", line 170, in init self.model.build_for_run ([ ( tf.float32, nn.get4Dshape (None,None,3) ) ]) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 154, in build_for_run self.run_output = self.call(self.run_placeholders) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in call return self.forward(*args, **kwargs) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\facelib\S3FDExtractor.py", line 94, in forward x = tf.nn.relu(self.conv1_2(x)) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\layers\LayerBase.py", line 14, in call return self.forward(*args, **kwargs) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\layers\Conv2D.py", line 97, in forward x = tf.pad (x, self.padding, mode='CONSTANT') File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper return target(*args, **kwargs) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\array_ops.py", line 3422, in pad result = gen_array_ops.pad(tensor, paddings, name=name) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 6484, in pad "Pad", input=input, paddings=paddings, name=name) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3536, in _create_op_internal op_def=op_def) File "K:\主要文件\DeepFaceLab_NVIDIA_build_12_22_2020\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1990, in init self._traceback = tf_stack.extract_stack()