OpenSeq2Seq icon indicating copy to clipboard operation
OpenSeq2Seq copied to clipboard

How much time take tacotron gst infererence?/infererence stuck

Open hadaev8 opened this issue 6 years ago • 6 comments

It stuck on

Successfully opened dynamic library libcublas.so.10.0

Code https://colab.research.google.com/drive/10DxRZTqmIPE6983fGEZ2UDjRrcvM99pK

hadaev8 avatar Aug 11 '19 20:08 hadaev8

Can you confirm that the code is running on GPU and not CPU?

blisc avatar Aug 15 '19 17:08 blisc

Should be gpu, but actually i'm not sure

hadaev8 avatar Aug 15 '19 17:08 hadaev8

Tacotron gst is known to hang on CPU #439. Although another user has reported errors on Google colab #476. Unfortunately, just like the CPU issue, we most likely will not be able to fix it if it is a Google colab issue.

blisc avatar Aug 15 '19 20:08 blisc

Same issue. It is running on local single GPU(1660 Ti) in mine and it is definitely using a GPU(nvidia_smi shows it's being used). I'm training a tacatron_float for text-to-speech conversion model on the LJSpeech dataset. I have tried reducing batch size but it doesn't seem to fix the issue.

JarvisStark007 avatar Aug 28 '19 12:08 JarvisStark007

Were you able to resolve the issue?I am also facing the same issue. I am using 6 GB Nvidia Geforce GTX 1060 with Cuda 10.0 and nvidia drivers 430.64. My batch size is 32. I had to reduce it from 48 due to cuda out of memory error. GPU utilisation in Nvidia-smi shows 1988/6078 MiB utilised. Volatile GPU util is very less (1%-4%)

smdshakeelhassan avatar May 14 '20 04:05 smdshakeelhassan

Use tf.GraphKeys.GLOBAL_VARIABLES instead. *** WARNING: Can't compute number of objects per step, since train model does not define get_num_objects_per_step method. 2020-08-04 14:37:19.226539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0

I am also facing the same issue. Any solution for the same? I am using tacotron_gst.py config file and pre-trained weights shared here (https://nvidia.github.io/OpenSeq2Seq/html/speech-synthesis.html). Other config is shared below- GPU- V100 Cuda - 10.0 Python - 3.6.10 Tensorflow - 1.15.0

conqueror7 avatar Aug 04 '20 14:08 conqueror7