Retrieval-based-Voice-Conversion-WebUI
Retrieval-based-Voice-Conversion-WebUI copied to clipboard
Published
20 hours ago •
RVC-Project
RVC-Project
Can't train model due to GPU limit which is wrong
Hi, I have successfully ran this app in a Docker container, however, once I get to train model buttons, I am getting this error:
/usr/local/lib/python3.10/site-packages/torch/functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:862.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
Process Process-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/train_nsf_sim_cache_sid_load_pretrain.py", line 223, in run
train_and_evaluate(
File "/app/train_nsf_sim_cache_sid_load_pretrain.py", line 436, in train_and_evaluate
scaler.scale(loss_disc).backward()
File "/usr/local/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/usr/local/lib/python3.10/site-packages/torch/autograd/__init__.py", line 200, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
The strange thing is, that script is using only half of my GPU memory, there's 50% more available, but for some reason it breaks.
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 3718 C /python3.10 N/A |
+---------------------------------------------------------------------------------------+
Fri Jul 14 23:43:45 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.06 Driver Version: 536.40 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3050 On | 00000000:01:00.0 On | N/A |
| 60% 45C P8 16W / 130W | 2016MiB / 8192MiB | 36% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 3718 C /python3.10 N/A |
+---------------------------------------------------------------------------------------+
Fri Jul 14 23:43:50 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.06 Driver Version: 536.40 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3050 On | 00000000:01:00.0 On | N/A |
| 60% 47C P2 34W / 130W | 4002MiB / 8192MiB | 61% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 3718 C /python3.10 N/A |
+---------------------------------------------------------------------------------------+
Fri Jul 14 23:43:55 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.06 Driver Version: 536.40 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3050 On | 00000000:01:00.0 On | N/A |
| 60% 45C P5 18W / 130W | 692MiB / 8192MiB | 15% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
