iperov
Training Fails using SAEHD
Ok, so when i entered through on default, it still fails. i dont understand the code, but it seems that it does not match model requirements? did i extract the images from video incorrectly? for both source and destination i did S3FD Full Face.... so it should be working correctly, if i understand. but it seems like i do not all the way. would like to get some help. thanks guys.
would be nice to get a updated tutorial on this as i dont see one anywhere.
Running trainer.
[new] No saved models found. Enter a name of a new model : Jax Jax
Model first run.
Choose one or several GPU idxs (separated by comma).
[CPU] : CPU [0] : GeForce RTX 2080 Ti
[0] Which GPU indexes to choose? : 0
[0] Autobackup every N hour ( 0..24 ?:help ) : 0 [n] Write preview history ( y/n ?:help ) : n [0] Target iteration : 0 [y] Flip faces randomly ( y/n ?:help ) : y [8] Batch_size ( ?:help ) : 8 [224] Resolution ( 64-512 ?:help ) : 224 [wf] Face type ( h/mf/f/wf ?:help ) : wf [dfhd] AE architecture ( df/liae/dfhd/liaehd/liaech ?:help ) : dfhd [256] AutoEncoder dimensions ( 32-1024 ?:help ) : 256 [64] Encoder dimensions ( 16-256 ?:help ) : 64 [64] Decoder dimensions ( 16-256 ?:help ) : 64 [22] Decoder mask dimensions ( 16-256 ?:help ) : 22 [y] Masked training ( y/n ?:help ) : y [y] Learn mask ( y/n ?:help ) : y [y] Eyes priority ( y/n ?:help ) : y [y] Place models and optimizer on GPU ( y/n ?:help ) : y [n] Use learning rate dropout ( y/n ?:help ) : n [y] Enable random warp of samples ( y/n ?:help ) : y [0.0] GAN power ( 0.0 .. 10.0 ?:help ) : 0.0 [0.0] 'True face' power. ( 0.0000 .. 1.0 ?:help ) : 0.0 [none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) : none [n] Enable gradient clipping ( y/n ?:help ) : n [n] Enable pretraining mode ( y/n ?:help ) : n Initializing models: 100%|###############################################################| 5/5 [00:11<00:00, 2.23s/it] Loading samples: 100%|#########################################################| 22498/22498 [00:21<00:00, 1070.36it/s] Loaded 0 packed faces from E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_dst\aligned Error: No training data provided. Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Trainer.py", line 55, in trainerThread debug=debug, File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\models\ModelBase.py", line 173, in init self.on_initialize() File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 549, in on_initialize generators_count=dst_generators_count ) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 46, in init raise ValueError('No training data provided.') ValueError: No training data provided. Process Process-59: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\19697_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "multiprocessing\process.py", line 258, in _bootstrap File "multiprocessing\process.py", line 93, in run File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func gen_data = next (self.generator_func) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 103, in batch_func raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) ) Exception: Exception occured in sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\19697_0.jpg. Error: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\19697_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
Process Process-61: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\07111_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "multiprocessing\process.py", line 258, in _bootstrap File "multiprocessing\process.py", line 93, in run File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func gen_data = next (self.generator_func) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 103, in batch_func raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) ) Exception: Exception occured in sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\07111_0.jpg. Error: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\07111_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
Process Process-60: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\01508_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "multiprocessing\process.py", line 258, in _bootstrap File "multiprocessing\process.py", line 93, in run File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func gen_data = next (self.generator_func) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 103, in batch_func raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) ) Exception: Exception occured in sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\01508_0.jpg. Error: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\01508_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
Process Process-62: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\18968_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "multiprocessing\process.py", line 258, in _bootstrap File "multiprocessing\process.py", line 93, in run File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func gen_data = next (self.generator_func) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 103, in batch_func raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) ) Exception: Exception occured in sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\18968_0.jpg. Error: Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 101, in batch_func x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 96, in process raise Exception ('sample %s type %s does not match model requirement %s. Consider extract necessary type of faces.' % (sample.filename, sample.face_type, face_type) ) Exception: sample E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA\workspace\data_src\aligned\18968_0.jpg type FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
"i did S3FD Full Face...." [wf] Face type ( h/mf/f/wf ?:help ) : wf FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
"i did S3FD Full Face...."
[wf] Face type ( h/mf/f/wf ?:help ) : wf
FaceType.FULL does not match model requirement FaceType.WHOLE_FACE. Consider extract necessary type of faces.
Hmmm, I see.
Ok, heres a update.
It says no training data provided. i did WF this time. i even did data_src util faceset enhance.... once i did that i then did wholeface S3FD.
After that, mas editor, and made corrections.... then i go into trainer and i get this error. i do understand that i set my resolution high, but it says no training data provided. that has nothing to do with resolution. i am lost now. dont know what i did wrong.
Running trainer.
[new] No saved models found. Enter a name of a new model : Jax McCleary Jax McCleary
Model first run.
Choose one or several GPU idxs (separated by comma).
[CPU] : CPU [0] : GeForce RTX 2080 Ti
[0] Which GPU indexes to choose? : 0
[0] Autobackup every N hour ( 0..24 ?:help ) : 0 [n] Write preview history ( y/n ?:help ) : n [0] Target iteration : 0 [y] Flip faces randomly ( y/n ?:help ) : y [8] Batch_size ( ?:help ) : 8 [512] Resolution ( 64-512 ?:help ) : 512 [wf] Face type ( h/mf/f/wf ?:help ) : wf [df] AE architecture ( df/liae/dfhd/liaehd/liaech ?:help ) : df [256] AutoEncoder dimensions ( 32-1024 ?:help ) : 256 [64] Encoder dimensions ( 16-256 ?:help ) : 64 [64] Decoder dimensions ( 16-256 ?:help ) : 64 [22] Decoder mask dimensions ( 16-256 ?:help ) : 22 [y] Masked training ( y/n ?:help ) : y [y] Learn mask ( y/n ?:help ) : y [y] Eyes priority ( y/n ?:help ) : y [y] Place models and optimizer on GPU ( y/n ?:help ) : y [n] Use learning rate dropout ( y/n ?:help ) : n [y] Enable random warp of samples ( y/n ?:help ) : y [0.0] GAN power ( 0.0 .. 10.0 ?:help ) : 0.0 [0.0] 'True face' power. ( 0.0000 .. 1.0 ?:help ) : 0.0 [none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) : none [n] Enable gradient clipping ( y/n ?:help ) : n [n] Enable pretraining mode ( y/n ?:help ) : n Initializing models: 100%|###############################################################| 5/5 [00:09<00:00, 1.85s/it] Loading samples: 100%|##########################################################| 22486/22486 [00:45<00:00, 493.47it/s] Loading samples: 0it [00:00, ?it/s] Error: No training data provided. Traceback (most recent call last): File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Trainer.py", line 55, in trainerThread debug=debug, File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\models\ModelBase.py", line 173, in init self.on_initialize() File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 549, in on_initialize generators_count=dst_generators_count ) File "E:\DEEP FAKE\SOFTWARE\DeepFaceLab\DeepFaceLab_NVIDIA_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 46, in init raise ValueError('No training data provided.') ValueError: No training data provided.
What this error says, is that it can't access the aligned folder in data_dst. It successfully loads 22486 from src, but fails when trying to fetch from dst. Usually means folders are renamed or not there.
What this error says, is that it can't access the aligned folder in data_dst. It successfully loads 22486 from src, but fails when trying to fetch from dst. Usually means folders are renamed or not there.
The folder is there, and I did not touch or rename it. It has not been moved. Is there any way around this? I already reinstalled the software twice...
I have found the solution. Or moreover the problem that was the root of it all.
From my little knowledge of the files and how it all works, I figured out that the reason it said that "no training model found" was because there was no images of the face (training model) in the folder called aligned. They were in Aligned confirmed.
Let me elaborate,
When I started the whole process, I went step by step. This is what I did, if you are curious.
Step 1: 2) extract images from video data_src
Step 2: 3) extract images from video data_dst FULL FPS
Step 3: data_stc extract whole_face S3FD
Step 4: I then opened up 4.1) data_src view aligned results to make sure they extracted properly and delete any random inaccuracies. Which I did so.
Step 5: I opened up 4.2) data_src util faceset enhance. Honestly don't know what that does, but i assumed it enhances the extractions, so I let it run. Took around 3 times longer that the src file took to extract.
Step 6: 5)data_dst extract whole_face S3FD
Step 7: 5.1)data_dst view aligned. Checked for inaccuracies and deleted the mistakes.
Step 8: 5.3) data_dst mask editor. This is where all the problems originate from my experience. I made new masks and corrected the faulty ones. Enough for it to train from. Or so I thought. By doing this it placed the images in Aligned confirmed folder. Keep that in mind for later.
Step 9: train SAEHD
Name: test Resolution: 128 Face_type: wf Models_opt_on_gpu: true Archetype: dfhd Ae_dims: 256 E_dims: 64 D_dims: 64 D_mask_dims: 22 Masked_training: true Learn_mask: true Eyes_priority: true Lr_dropout: false Random_warp: true Gain_power: 0.0 True_face_power: 0.0 Face_style_power: 0.0 Bg_style_power: 0.0 Ct_mode: none Clipgraddient: false Pretrain: false Autobackup_hour: 0 Write_preview_history: false Target_iter: 500,000 Rabdom_flip: true Batch_size: 8 Running on RTX 2080 TI VRAM: 11gb
Specs and settings aside, the fact that the images were in aligned confined was the reason that there was no training model available. The images were supposed to stay in aligned folder.
Also if anyone is willing to explain the different architecture, I would be great full.
i think it is important to note that i am on build version 03_07_2020
Below is a preview of my main folder layout. along with the respective dst folder and how it looks.
Yeah that would be enough to get the error. If it's not in the correct path data_dst/aligned it cant access it. Why are you on that old and very outdated build? Get yourself the last(04.01) and fire up a df-ud 320 wf model. Optimization for nvidia cards have traveled light years since then. Description of the different architectures you'll find in the guides on mrdeepfakes.com. Together with pre-trained models and everything else needed.
I got the following error while training the model, I'm new to this, please comment below for any suggestions.
Running trainer.
[new] No saved models found. Enter a name of a new model : new
Model first run.
Choose one or several GPU idxs (separated by comma).
[CPU] : CPU [0] : GeForce GTX 1650
[0] Which GPU indexes to choose? : 0
[0] Autobackup every N hour ( 0..24 ?:help ) :
0
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[y] Flip faces randomly ( y/n ?:help ) :
y
[8] Batch_size ( ?:help ) :
8
[128] Resolution ( 64-640 ?:help ) :
128
[f] Face type ( h/mf/f/wf/head ?:help ) : wf
wf
[liae-ud] AE architecture ( ?:help ) :
liae-ud
[256] AutoEncoder dimensions ( 32-1024 ?:help ) :
256
[64] Encoder dimensions ( 16-256 ?:help ) :
64
[64] Decoder dimensions ( 16-256 ?:help ) :
64
[22] Decoder mask dimensions ( 16-256 ?:help ) :
22
[y] Masked training ( y/n ?:help ) :
y
[n] Eyes and mouth priority ( y/n ?:help ) : y
[n] Uniform yaw distribution of samples ( y/n ?:help ) :
n
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[n] Use learning rate dropout ( n/y/cpu ?:help ) :
n
[y] Enable random warp of samples ( y/n ?:help ) :
y
[0.0] GAN power ( 0.0 .. 1.0 ?:help ) :
0.0
[0.0] Face style power ( 0.0..100.0 ?:help ) :
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) :
0.0
[none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
none
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 60%|#####################################8 | 3/5 [00:21<00:14, 7.35s/it]Traceback (most recent call last):
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier\_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) _pywrap_tensorflow_internal = swig_import_helper()
File "imp.py", line 243, in load_module File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper File "imp.py", line 343, in load_dynamic _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)ImportError : File "imp.py", line 243, in load_module DLL load failed: The paging file is too small for this operation to complete. File "imp.py", line 343, in load_dynamic
During handling of the above exception, another exception occurred:
ImportErrorTraceback (most recent call last):
: File "
During handling of the above exception, another exception occurred:
File "multiprocessing\spawn.py", line 105, in spawn_main
Traceback (most recent call last):
File "multiprocessing\spawn.py", line 115, in main
File "
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python_init_.py", line 49, in
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in
ImportError File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.
ImportError: Traceback (most recent call last):
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
File "C:\Mazhar\Misc\DFL\DeepFaceLab\DeepFaceLab_NVIDIA_RTX2080Ti_and_earlier_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
File "
A few things I noticed while going through your error message.
For your actual problem, you are running out of VRAM. (Memory) you can see the message yourself at the very bottom of the error code. To meditate this, you need to reduce your resolution from default of 128, down to something smaller. Try 64.
Another thing I saw while looking through your settings is that you have 0 set for iterations. Set that to something like 150,000. I'm not sure if 0 means infinite until you stop it manually. But in my case I could not stop it when pressing enter so I set a limit of 500k for myself.
Another thing is if you don't do wf (whole face) not while head at extraction, then you'll also get a error. It'll say no training data.
Anyways. For now, try to do the things I mentioned above and let me know how it goes.
I also have error messages training with SAEHD, please I need help. Below is the error message I received
debug=debug)
File "C:\Users\Jotham\Desktop\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\ModelBase.py", line 193, in init
self.on_initialize()
File "C:\Users\Jotham\Desktop\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 341, in on_initialize
self.src_dst_opt.initialize_variables (self.src_dst_saveable_weights, vars_on_cpu=optimizer_vars_on_cpu, lr_dropout_on_cpu=self.options['lr_dropout']=='cpu')
File "C:\Users\Jotham\Desktop\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 38, in initialize_variables
vs = { v.name : tf.get_variable ( f'vs_{v.name}'.replace(':',''), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
File "C:\Users\Jotham\Desktop\DeepFaceLab_NVIDIA_RTX3000_series_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 38, in
Recently encountered this type of problem does any know what to do ?
Error: invalid load key, '\x00'. Traceback (most recent call last): File "E:\DF Main\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread debug=debug) File "E:\DF Main\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\ModelBase.py", line 193, in init self.on_initialize() File "E:\DF Main\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 654, in on_initialize do_init = not model.load_weights( self.get_strpath_storage_for_file(filename) ) File "E:\DF Main\DeepFaceLab_NVIDIA_up_to_RTX2080Ti_internal\DeepFaceLab\core\leras\layers\Saveable.py", line 72, in load_weights d = pickle.loads(d_dumped) _pickle.UnpicklingError: invalid load key, '\x00'
