jaywalnut310
Training on Tesla K80
Hi, Using Tesla K80 to train the model is giving the following error. Does the model require specific GPU architecture for training?
File "train.py", line 290, in
-- Process 0 terminated with the following error: Traceback (most recent call last): File "/anaconda/envs/vits/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap fn(i, *args) File "/media/hdd1tb/tts-VITS/vits-main/train.py", line 117, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "/media/hdd1tb/tts-VITS/vits-main/train.py", line 162, in train_and_evaluate hps.data.mel_fmax File "/media/hdd1tb/tts-VITS/vits-main/mel_processing.py", line 105, in mel_spectrogram_torch center=center, pad_mode='reflect', normalized=False, onesided=True) File "/anaconda/envs/vits/lib/python3.7/site-packages/torch/functional.py", line 465, in stft return _VF.stft(input, n_fft, hop_length, win_length, window, normalized, onesided) RuntimeError: cuFFT doesn't support signals of half type with compute capability less than SM_53, but the device containing input half tensor only has SM_37
You cannot train vits on a K80. K80's are a very weak GPU. You need at least a Tesla P100 or a T4 in order to avoid errors when training. A valid explanation for this is because K80's don't have enough memory for training.
You cannot train vits on a K80. K80's are a very weak GPU. You need at least a Tesla P100 or a T4 in order to avoid errors when training. A valid explanation for this is because K80's don't have enough memory for training.
That's not true. You can train on any GPU which supports cuda, but have to set fitting batch size. It reduce resulting quality, true, but it doesn't mean "you cannot train".