DeepFaceLab Exception while trying to train SAEHD

OS: Windows 11 x64 DFL Version: DeepFaceLab_NVIDIA_RTX3000_series_build_11_20_2021 GPU: nVidia GeForce RTX 3090 FE CPU: AMD Ryzen Threadripper 3960x RAM: 32G

After extracting faces of src and dst, I tried to train using SAEHD. I have RTX 2080 Super and RTX 3090 plugged on to my motherboard, but I use RTX 3090 only. Let me paste the trace below to refer the configs I have provided and the exception;

Running trainer.

[new] No saved models found. Enter a name of a new model : test_saehd
test_saehd

Model first run.

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce RTX 3090
  [1] : NVIDIA GeForce RTX 2080 SUPER

[0] Which GPU indexes to choose? : 0
0

[1] Autobackup every N hour ( 0..24 ?:help ) : 1
1
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[n] Flip SRC faces randomly ( y/n ?:help ) :
n
[y] Flip DST faces randomly ( y/n ?:help ) :
y
[8] Batch_size ( ?:help ) :
8
[128] Resolution ( 64-640 ?:help ) :
128
[f] Face type ( h/mf/f/wf/head ?:help ) :
f
[liae-ud] AE architecture ( ?:help ) :
liae-ud
[256] AutoEncoder dimensions ( 32-1024 ?:help ) :
256
[64] Encoder dimensions ( 16-256 ?:help ) :
64
[64] Decoder dimensions ( 16-256 ?:help ) :
64
[22] Decoder mask dimensions ( 16-256 ?:help ) :
22
[n] Eyes and mouth priority ( y/n ?:help ) :
n
[n] Uniform yaw distribution of samples ( y/n ?:help ) :
n
[n] Blur out mask ( y/n ?:help ) :
n
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[n] Use learning rate dropout ( n/y/cpu ?:help ) :
n
[y] Enable random warp of samples ( y/n ?:help ) :
y
[0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
0.0
[0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
0.0
[0.0] Face style power ( 0.0..100.0 ?:help ) :
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) :
0.0
[none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
none
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 100%|###############################################################| 5/5 [00:01<00:00,  3.42it/s]
Loading samples: 100%|############################################################| 4270/4270 [00:08<00:00, 496.97it/s]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "multiprocessing\spawn.py", line 105, in spawn_main
Exception in thread Thread-25:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
  File "multiprocessing\spawn.py", line 115, in _main

  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\samplelib\__init__.py", line 1, in <module>
    from .Sample import Sample
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\samplelib\Sample.py", line 7, in <module>
    from core.cv2ex import *
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\cv2ex.py", line 5, in <module>
    from core import imagelib
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\imagelib\__init__.py", line 5, in <module>
    from .text import get_text_image, get_draw_text_lines
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\imagelib\text.py", line 3, in <module>
    from PIL import Image, ImageDraw, ImageFont
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 764, in get_code
  File "<frozen importlib._bootstrap_external>", line 833, in get_data
MemoryError

Somebody please tell me why this is happening and how to fix this?

the entire error trace is like below;

Running trainer.

[new] No saved models found. Enter a name of a new model : test_saehd
test_saehd

Model first run.

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce RTX 3090
  [1] : NVIDIA GeForce RTX 2080 SUPER

[0] Which GPU indexes to choose? : 0
0

[1] Autobackup every N hour ( 0..24 ?:help ) : 1
1
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[n] Flip SRC faces randomly ( y/n ?:help ) :
n
[y] Flip DST faces randomly ( y/n ?:help ) :
y
[8] Batch_size ( ?:help ) :
8
[128] Resolution ( 64-640 ?:help ) :
128
[f] Face type ( h/mf/f/wf/head ?:help ) :
f
[liae-ud] AE architecture ( ?:help ) :
liae-ud
[256] AutoEncoder dimensions ( 32-1024 ?:help ) :
256
[64] Encoder dimensions ( 16-256 ?:help ) :
64
[64] Decoder dimensions ( 16-256 ?:help ) :
64
[22] Decoder mask dimensions ( 16-256 ?:help ) :
22
[n] Eyes and mouth priority ( y/n ?:help ) :
n
[n] Uniform yaw distribution of samples ( y/n ?:help ) :
n
[n] Blur out mask ( y/n ?:help ) :
n
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[n] Use learning rate dropout ( n/y/cpu ?:help ) :
n
[y] Enable random warp of samples ( y/n ?:help ) :
y
[0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
0.0
[0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
0.0
[0.0] Face style power ( 0.0..100.0 ?:help ) :
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) :
0.0
[none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
none
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 100%|###############################################################| 5/5 [00:01<00:00,  3.42it/s]
Loading samples: 100%|############################################################| 4270/4270 [00:08<00:00, 496.97it/s]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "multiprocessing\spawn.py", line 105, in spawn_main
Exception in thread Thread-25:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
  File "multiprocessing\spawn.py", line 115, in _main

  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\samplelib\__init__.py", line 1, in <module>
    from .Sample import Sample
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\samplelib\Sample.py", line 7, in <module>
    from core.cv2ex import *
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\cv2ex.py", line 5, in <module>
    from core import imagelib
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\imagelib\__init__.py", line 5, in <module>
    from .text import get_text_image, get_draw_text_lines
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\imagelib\text.py", line 3, in <module>
    from PIL import Image, ImageDraw, ImageFont
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 764, in get_code
  File "<frozen importlib._bootstrap_external>", line 833, in get_data
MemoryError
Exception in thread Thread-22:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-13:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-10:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-18:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-21:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-27:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-14:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-15:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-12:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-16:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-23:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-6:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-24:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-7:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-26:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-20:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-9:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-5:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-19:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-8:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-17:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-11:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Exception in thread Thread-4:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
    generator._start()
  File "C:\Users\User\Documents\DF\DFLab\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
    p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

Feb 07 '22 10:02 dflguru

Hi, did you ever find a fix for this? I have the exact same issue. Thanks

Jun 15 '22 20:06 prehuman

Hi, I encountered the (somewhat) same issue while training the SAEHD model. I think this issue might be related to large thread sizes, because I'm training on an AMD Epyc processor with 24 cores and @dflguru is using a threadripper on his side. Another machine of mine with the same configuration does work totally fine. I'm not sure why my second machine doesn't want to train. Anyway, a small fix to start training at least is to manually reduce the amount of cores the SAEHD model uses while training. This can be done by replacing multiprocessing.cpu_count() in line 669 of \DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\models\Model_SAEHD\Model.py with a fixed number e.g. 8 to use only 8 cores. This helped at least to get the training started. I'm not quite sure if there is a fixed limit on multiprocessing... Also some more dynamic approaches like reducing the cpu_count by one ( multiprocessing.cpu_count() -1) didn't seem to work. More investigation is needed, but this could be a first starting point.

Sep 30 '22 12:09 cglukas

@cglukas Thank You! This issue was driving me mad. Both my dual xeon and threadripper servers were crashing while my laptop was fine. Reducing the cpu_count works like a charm!

Mar 27 '23 00:03 GeTelis

Issue solved / already answered (or it seems like user error), please close it.

Jun 08 '23 22:06 joolstorrentecalo

Hey @JDTCC , I don't think that the issue is resolved. The current solution is to go into the code and manually set the available threads for the multiprocessing package. IMHO, this should be somehow implemented into the public code. Maybe with a try/catch statement to reduce the threads when the broken pipe error occurs. Or with an extra setting to limit the amount of cores.

Jun 09 '23 05:06 cglukas