"GET was unable to find an engine to execute this computation" during One-Click-Training

[nicola-spreafico]

Hello, I downloaded the portable-app from https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/RVC0813Nvidia.7z I have an audio sample with my voice with duration of 15 minutes.

The preprocess-step went fine

start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\\voice-ai\\ciardelli-audio-sample', '40000', '6', 'C:\\voice-ai\\RVC0813Nvidia/logs/cia, 'False']
C:\voice-ai\cia-audio-sample/cia-sample.wav->Suc.
end preprocess

The extraction went fine

['extract_f0_rmvpe.py', '2', '0', '0', 'C:\\voice-ai\\RVC0813Nvidia/logs/cia', 'True']
['extract_f0_rmvpe.py', '2', '1', '0', 'C:\\voice-ai\\RVC0813Nvidia/logs/cia', 'True']
todo-f0-138
f0ing,now-0,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_0.wav
todo-f0-138
f0ing,now-0,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_1.wav
f0ing,now-27,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_157.wav
f0ing,now-27,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_158.wav
f0ing,now-54,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_216.wav
f0ing,now-54,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_217.wav
f0ing,now-81,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_278.wav
f0ing,now-81,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_279.wav
f0ing,now-108,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_340.wav
f0ing,now-108,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_34.wav
f0ing,now-135,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_93.wav
f0ing,now-135,all-138,-C:\voice-ai\RVC0813Nvidia/logs/cia/1_16k_wavs/0_92.wav
['extract_feature_print.py', 'cuda:0', '1', '0', '0', 'C:\\voice-ai\\RVC0813Nvidia/logs/cia', 'v2']
C:\voice-ai\RVC0813Nvidia/logs/cia
load model(s) from hubert_base.pt
move model to cuda
all-feature-276
now-276,all-0,0_0.wav,(149, 768)
now-276,all-27,0_125.wav,(149, 768)
now-276,all-54,0_157.wav,(149, 768)
now-276,all-81,0_189.wav,(149, 768)
now-276,all-108,0_216.wav,(149, 768)
now-276,all-135,0_247.wav,(149, 768)
now-276,all-162,0_278.wav,(146, 768)
now-276,all-189,0_309.wav,(101, 768)
now-276,all-216,0_34.wav,(149, 768)
now-276,all-243,0_6.wav,(59, 768)
now-276,all-270,0_92.wav,(149, 768)
all-feature-done

When using the One-Click-Training button this is the output in web console

Passaggio 1: elaborazione dei dati
runtime\python.exe trainset_preprocess_pipeline_print.py "C:\voice-ai\cia-audio-sample" 40000 4 "C:\voice-ai\RVC0813Nvidia/logs/cia" False
step2a:正在提取音高
runtime\python.exe extract_f0_rmvpe.py 2 0 0 "C:\voice-ai\RVC0813Nvidia/logs/cia" True 
runtime\python.exe extract_f0_rmvpe.py 2 1 0 "C:\voice-ai\RVC0813Nvidia/logs/cia" True 
Passaggio 2b: estrazione delle funzionalità
runtime\python.exe extract_feature_print.py cuda:0 1 0 0 "C:\voice-ai\RVC0813Nvidia/logs/cia" v2
Passaggio 3a: è iniziato l'addestramento del modello
write filelist done
runtime\python.exe train_nsf_sim_cache_sid_load_pretrain.py -e "cia" -sr 40k -f0 1 -bs 2 -g 0 -te 10 -se 5 -pg pretrained_v2/f0G40k.pth -pd pretrained_v2/f0D40k.pth -l 1 -c 0 -sw 1 -v v2
Addestramento completato. 
(38578, 768),989
training index
adding index
成功构建索引, added_IVF989_Flat_nprobe_1_cia_v2.index
Tutti i processi sono stati completati!

But the truth is that in the prompt console where I launched the application I'm getting this error

INFO:cia:{'train': {'log_interval': 200, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 2, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 12800, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'max_wav_value': 32768.0, 'sampling_rate': 40000, 'filter_length': 2048, 'hop_length': 400, 'win_length': 2048, 'n_mel_channels': 125, 'mel_fmin': 0.0, 'mel_fmax': None, 'training_files': './logs\\cia/filelist.txt'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [10, 10, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'use_spectral_norm': False, 'gin_channels': 256, 'spk_embed_dim': 109}, 'model_dir': './logs\\cia', 'experiment_dir': './logs\\cia', 'save_every_epoch': 5, 'name': 'cia', 'total_epoch': 10, 'pretrainG': 'pretrained_v2/f0G40k.pth', 'pretrainD': 'pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 1, 'save_every_weights': '1', 'if_cache_data_in_gpu': 0}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
gin_channels: 256 self.spk_embed_dim: 109
INFO:cia:loaded pretrained pretrained_v2/f0G40k.pth
<All keys matched successfully>
INFO:cia:loaded pretrained pretrained_v2/f0D40k.pth
<All keys matched successfully>
C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
Process Process-1:
Traceback (most recent call last):
  File "multiprocessing\process.py", line 315, in _bootstrap
  File "multiprocessing\process.py", line 108, in run
  File "C:\voice-ai\RVC0813Nvidia\train_nsf_sim_cache_sid_load_pretrain.py", line 228, in run
    train_and_evaluate(
  File "C:\voice-ai\RVC0813Nvidia\train_nsf_sim_cache_sid_load_pretrain.py", line 441, in train_and_evaluate
    scaler.scale(loss_disc).backward()
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\torch\autograd\__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: GET was unable to find an engine to execute this computation

I also tried to do a model inference with an already bult-in model, but even there I'm getting an errore, and in the console I can see the error:

RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


Traceback (most recent call last):
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\blocks.py", line 1007, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\components.py", line 2076, in postprocess
    processing_utils.audio_to_file(sample_rate, data, file.name)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
    data = convert_to_16_bit_wav(data)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
    if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'
Traceback (most recent call last):
  File "C:\voice-ai\RVC0813Nvidia\infer-web.py", line 211, in vc_single
    audio_opt = vc.pipeline(
  File "C:\voice-ai\RVC0813Nvidia\vc_infer_pipeline.py", line 342, in pipeline
    sid = torch.tensor(sid, device=self.device).unsqueeze(0).long()
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


Traceback (most recent call last):
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\blocks.py", line 1007, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\components.py", line 2076, in postprocess
    processing_utils.audio_to_file(sample_rate, data, file.name)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
    data = convert_to_16_bit_wav(data)
  File "C:\voice-ai\RVC0813Nvidia\runtime\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
    if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'

Can you help me understand how can I solve this issue?

I'm running on Windows 11 22H2 I Have NVIDIA GeForce GTX 970

Thank you

[RVC-Boss]

GTX970 is too old to run RVC orzzzzzz

[RVC-Boss]

Maybe you should change the version of pytorch

[nicola-spreafico]

GTX970 is too old to run RVC orzzzzzz

Ok thank you, I will look for another PC. For the moment I used a virtual machine on public cloud with "1 x NVIDIA Tesla P100" GPU and it all went fine

[chyueG]

gtx3090 also has the same problem

[maozj6]

It is the web ui issue. The GET cannot reach to run

The issue can be solved by directly run infer/modules/train/extract_feature_print.py

remember to set parameters for ['extract_f0_rmvpe.py', '2', '0', '0', 'C:\voice-ai\RVC0813Nvidia/logs/cia', 'True']: device = sys.argv[1] n_part = int(sys.argv[2]) i_part = int(sys.argv[3]) ...

[iwoolf]

I have the same problem in linux on rtx3060 File "/media/iwoolf/BigDrive/anaconda3/envs/rbvc-webui/lib/python3.10/site-packages/torch/autograd/init.py", line 266, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: GET was unable to find an engine to execute this computation