tensorflow-wavenet
tensorflow-wavenet copied to clipboard
ibab
Training tensorflow-wavenet
What are the time scales for training wavenet? testing? Did someone try it on AWS? What were the specs of the cloud?
I am running it on paperspace (GPU enabled, let me know if you want to be more specific, but believe me, this virtual machine is a beast). Took 2,5 days to train with 1e5 (100000 steps). To generate 3 minutes of audio takes like 6 hours or something. Result: a sort of average turd. I am a beginner in coding and one must tweak this script somehow.
I am running it on paperspace (GPU enabled, let me know if you want to be more specific, but believe me, this virtual machine is a beast). Took 2,5 days to train with 1e5 (100000 steps). To generate 3 minutes of audio takes like 6 hours or something. Result: a sort of average turd. I am a beginner in coding and one must tweak this script somehow.
Could you please tell me how you generate and get the voice? My training steps are only 10,000 and it takes 2 days, because 100,000 steps are expected to take more than ten days. Is it because my training time and step size are insufficient that the speech generated by running "generator.py" is 0 seconds?
You have to use gpu for this, cpu is too slow.
If that is not the problem, try changing the training parameters. Hope it helps.
Op vr 14 feb. 2020 04:10 schreef Ahapy [email protected]:
I am running it on paperspace (GPU enabled, let me know if you want to be more specific, but believe me, this virtual machine is a beast). Took 2,5 days to train with 1e5 (100000 steps). To generate 3 minutes of audio takes like 6 hours or something. Result: a sort of average turd. I am a beginner in coding and one must tweak this script somehow.
Could you please tell me how you generate and get the voice? My training steps are only 10,000 and it takes 2 days, because 100,000 steps are expected to take more than ten days. Is it because my training time and step size are insufficient that the speech generated by running "generator.py" is 0 seconds?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ibab/tensorflow-wavenet/issues/345?email_source=notifications&email_token=AIWQVUTBXXZRFMWBGSRDJ7LRCYDT3A5CNFSM4FFAPUM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELXNHJA#issuecomment-586077092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWQVUSQX5IID6DE5IQLQMDRCYDT3ANCNFSM4FFAPUMQ .
You have to use gpu for this, cpu is too slow. If that is not the problem, try changing the training parameters. Hope it helps. Op vr 14 feb. 2020 04:10 schreef Ahapy [email protected]: … I am running it on paperspace (GPU enabled, let me know if you want to be more specific, but believe me, this virtual machine is a beast). Took 2,5 days to train with 1e5 (100000 steps). To generate 3 minutes of audio takes like 6 hours or something. Result: a sort of average turd. I am a beginner in coding and one must tweak this script somehow. Could you please tell me how you generate and get the voice? My training steps are only 10,000 and it takes 2 days, because 100,000 steps are expected to take more than ten days. Is it because my training time and step size are insufficient that the speech generated by running "generator.py" is 0 seconds? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#345?email_source=notifications&email_token=AIWQVUTBXXZRFMWBGSRDJ7LRCYDT3A5CNFSM4FFAPUM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELXNHJA#issuecomment-586077092>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWQVUSQX5IID6DE5IQLQMDRCYDT3ANCNFSM4FFAPUMQ .
Thanks for your suggestions, but I'm sure I have used a GPU.
When running "train.py", there was such a paragraph:
2020-02-14 17:09:16.715677: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats: Limit: 1422723891 InUse: 1344603648 MaxInUse: 1416595456 NumAllocs: 2844 MaxAllocSize: 193528576
And I tested these two lines of codefrom tensorflow.python.client import device_lib print(device_lib.list_local_devices())to detect whether the GPU or CPU is used at runtime.
The results are shown below:
[name: "/device:CPU:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 8034237779902744267 , name: "/device:GPU:0" device_type: "GPU" memory_limit: 1422723891 locality { bus_id: 1 links { } } incarnation: 4308487621035411117 physical_device_desc: "device: 0, name: GeForce MX150, pci bus id: 0000:01:00.0, compute capability: 6.1" ]
Limit: 1422723891 in "train.py" refers to device_type: "GPU" memory_limit: 1422723891 in the test results, so I think "train.py" uses a GPU instead of a CPU.