WaveRNN icon indicating copy to clipboard operation
WaveRNN copied to clipboard

Published 20 hours ago • verified publisher icon fatchord

how long did you train the WaveRNN model

Open li-xx-5 opened this issue 5 years ago • 19 comments

hello,i have been training the waveRNN model.However, the training speed of this model remains at 0.1step/s.Is there anythong wrong?thank you

li-xx-5 avatar May 23 '19 07:05 li-xx-5

@doctor-xiang Something is wrong if you are only getting 0.1 steps/s - normally I get around 3 steps/s on a GTX1080. What GPU and pytorch version are you running?

fatchord avatar May 23 '19 08:05 fatchord

@fatchord ,i use the TITAN 1080.But the batch size of my model is 32,so i get that rate.If I reduce the batch size, the voice quality will decline. What is the best batch size for training this model ? Thank you

li-xx-5 avatar May 23 '19 08:05 li-xx-5

@doctor-xiang I'm getting 3 steps/s for a batch size of 32. I've found that a batch size of 16 or 32 works well (I haven't experimented outside of that). What version of pytorch are you using?

fatchord avatar May 23 '19 09:05 fatchord

@fatchord the version of 1.0.0

li-xx-5 avatar May 23 '19 11:05 li-xx-5

@fatchord i had solved that.thank you

li-xx-5 avatar May 24 '19 02:05 li-xx-5

@doctor-xiang Great to hear! Just curious what happened? In case anyone else runs into the same problem.

fatchord avatar May 24 '19 06:05 fatchord

@fatchord There was an impact between tasks.

li-xx-5 avatar May 25 '19 02:05 li-xx-5

@fatchord Hello, may i ask what is the speed of speech synthesis when you use GTX1080.

li-xx-5 avatar May 27 '19 00:05 li-xx-5

@fatchord There was an impact between tasks.

hello, I met the same problem as you, and the training speed of wavernn only 0.3step/s , how did you fix it? Thank you!

kanatazhan avatar Jun 20 '19 04:06 kanatazhan

@kanatazhan you might want @doctor-xiang to get his attention as I've no idea how he solved the problem :)

fatchord avatar Jun 20 '19 08:06 fatchord

@kanatazhan There may be multiple tasks running that take up resources on the GPU, resulting in slow speeds.

1105060120 avatar Jun 26 '19 01:06 1105060120

Using a K80 GPU and batch size of 32 I have about 0.8step/s, and the GPU seems to be around 90% utilisation

maelp avatar Aug 06 '19 14:08 maelp

I'm using sample_rate=22050 so perhaps this could explain some difference in the speed

maelp avatar Aug 06 '19 14:08 maelp

@maelp your sample_rate is fine. I wonder - are you accidentally using cpu? Can you try watch nvidia-smi and run training - does the mem and gpu-util increase when starting off?

fatchord avatar Aug 06 '19 15:08 fatchord

I'm using the GPU, it shows the use as close to 90% so I think it is working

maelp avatar Aug 06 '19 16:08 maelp

And python3.7 and pytorch 1.0

maelp avatar Aug 06 '19 16:08 maelp

K80s are pretty ancient gpus, ~and almost entirely catered to double precision floating point operations~. To be honest, I am not surprised by this training rate. A simple consumer level (GTX 1070) graphics card will blow it away

TheButlah avatar Aug 08 '19 16:08 TheButlah

I'm getting 2.3 steps/s on an RTX 3080. CUDA GPU usage hovers around 50%. Is that right?

serg06 avatar Oct 29 '20 22:10 serg06

I'm facing the same issue. Can anyone explain this issue clearly?

TinBarbie avatar Mar 02 '21 04:03 TinBarbie