iperov
Deepfacelab gives error [Error: Attempting to use uninitialized value conv2d_1/bias] when running trainer
Here's the code. Please help me realize what I have to fix. I am just learning the software and not familiar with code language so please be patient with me. I just want to know if this is something to do with my file placement, a software I don't have on my computer or if it just something my computer simply cannot do. Thank you!
Running trainer.
Loading model...
Model first run. Enable autobackup? (y/n ?:help skip:n) : y Write preview history? (y/n ?:help skip:n) : n Target iteration (skip:unlimited/default) : 0 Batch_size (?:help skip:0) : 20 Flip faces randomly? (y/n ?:help skip:y) : y Resolution ( 64-256 ?:help skip:128) : 128 Half or Full face? (h/f, ?:help skip:f) : f Learn mask? (y/n, ?:help skip:y ) : y Optimizer mode? ( 1,2,3 ?:help skip:1) : 1 AE architecture (df, liae ?:help skip:df) : df AutoEncoder dims (32-1024 ?:help skip:512) : 512 Encoder dims per channel (21-85 ?:help skip:42) : 42 Decoder dims per channel (10-85 ?:help skip:21) : 21 Use CA weights? (y/n, ?:help skip:n ) : n Use pixel loss? (y/n, ?:help skip:n ) : n Face style power ( 0.0 .. 100.0 ?:help skip:0.00) : 0.0 Background style power ( 0.0 .. 100.0 ?:help skip:0.00) : 0.0 Color transfer mode apply to src faceset. ( none/rct/lct/mkl/idt/sot, ?:help skip:none) : none Enable gradient clipping? (y/n, ?:help skip:n) : y Pretrain the model? (y/n, ?:help skip:n) : n Using TensorFlow backend. Loading: 100%|####################################################################| 6985/6985 [01:04<00:00, 109.09it/s] Loading: 100%|#######################################################################| 159/159 [00:01<00:00, 85.12it/s] ========== Model Summary =========== == == == Model name: SAE == == == == Current iteration: 0 == == == ==-------- Model Options ---------== == == == autobackup: True == == random_flip: True == == resolution: 128 == == face_type: f == == learn_mask: True == == optimizer_mode: 1 == == archi: df == == ae_dims: 512 == == e_ch_dims: 42 == == d_ch_dims: 21 == == ca_weights: False == == pixel_loss: False == == face_style_power: 0.0 == == bg_style_power: 0.0 == == ct_mode: none == == clipgrad: True == == batch_size: 20 == == == ==---------- Running On ----------== == == == Device index: 0 == == Name: Quadro M620 == == VRAM: 2.00GB == == ==
/!
/!\ WARNING:
/!\ You are using a GPU with 2GB or less VRAM. This may significantly reduce the quality of your result!
/!\ If training does not start, close all programs and try again.
/!\ Also you can disable Windows Aero Desktop to increase available VRAM.
/!
Error: Attempting to use uninitialized value conv2d_1/bias
[[{{node conv2d_1/bias/_856}} = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6_conv2d_1/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[{{node conv2d_1/bias/_857}} = _Recv_start_time=0, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6_conv2d_1/bias", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Traceback (most recent call last):
File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value conv2d_1/bias
[[{{node conv2d_1/bias/_856}} = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6_conv2d_1/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[{{node conv2d_1/bias/_857}} = _Recv_start_time=0, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6_conv2d_1/bias", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\mainscripts\Trainer.py", line 76, in trainerThread model_save() File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\mainscripts\Trainer.py", line 62, in model_save model.save() File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\models\ModelBase.py", line 349, in save self.onSave() File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\models\Model_SAE\Model.py", line 497, in onSave self.save_weights_safe( self.get_model_filename_list() ) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\models\ModelBase.py", line 451, in save_weights_safe model.save_weights( filename) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\keras\engine\network.py", line 1121, in save_weights saving.save_weights_to_hdf5_group(f, self.layers) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\keras\engine\saving.py", line 572, in save_weights_to_hdf5_group weight_values = K.batch_get_value(symbolic_weights) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2420, in batch_get_value return get_session().run(ops) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 929, in run run_metadata_ptr) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run run_metadata) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value conv2d_1/bias [[{{node conv2d_1/bias/_856}} = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6_conv2d_1/bias", _device="/job:localhost/replica:0/task:0/device:GPU:0"]] [[{{node conv2d_1/bias/_857}} = _Recv_start_time=0, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6_conv2d_1/bias", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Set each and every setting to the lowest possible, and turn off every setting that is possible to turn off.
See if you can get any progress.... Then try each setting at its default value.
Note:
batch_size: 20 seems high. Try the default (4)
AutoEncoder dims: 512 seems high. Try the default or 128
Sorry for the late reply. Tried your suggested settings and got this error message
Error: OOM when allocating tensor with shape[4,132,132,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node model_1_1/zero_padding2d_1/Pad}} = Pad[T=DT_FLOAT, Tpaddings=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_input_7_0_1/_1069, model_1/zero_padding2d_1/Pad/paddings)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node Mean_3/_1109}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7444_Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Traceback (most recent call last): File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\mainscripts\Trainer.py", line 111, in trainerThread iter, iter_time = model.train_one_iter() File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\models\ModelBase.py", line 507, in train_one_iter losses = self.onTrainOneIter(sample, self.generator_list) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\DeepFaceLab\models\Model_SAE\Model.py", line 506, in onTrainOneIter src_loss, dst_loss, = self.src_dst_train (feed) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in call return self._call(inputs) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1439, in call run_metadata_ptr) File "C:\Users\shado\Desktop\Deepfake maybe\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4,132,132,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node model_1_1/zero_padding2d_1/Pad}} = Pad[T=DT_FLOAT, Tpaddings=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_input_7_0_1/_1069, model_1/zero_padding2d_1/Pad/paddings)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node Mean_3/_1109}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7444_Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Did you ever find the answer?
I gave up on it assuming that my computer just didn't have enough power to run it. I just recently got a new computer with better specs and maybe that is the difference that was needed. I may try again soon, but maybe do research and see if your computer can handle the program.