deep-voice-conversion
deep-voice-conversion copied to clipboard
andabi
May I ask your hardware?
Hi, may I ask about your hardware specification? Such as which graphics card you use? How many ram your pc have? thanks.
@chikiuso I used a server which has 8 Nvidia Tesla P40 GPU and 200GB memory. But single GPU core was enough.
@andabi
I ran your code on a server like this:
- 8 Nvidia GTX1080
- about 40G graphics memory
- 200GB of memory
but the progress will always stop on " Creating TensorFlow device" and don't show any more infomation.
@HudsonHuang Please check the paths(data_path or something) in hparam.py again. If you set the paths incorrectly, the problem you mentioned could happen.
I use: name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845 pciBusID: 0000:01:00.0 totalMemory: 976.12MiB freeMemory: 948.88MiB
Meet the similar issue, the progress will always stop on " Creating TensorFlow device" and don't show any more information, when I installed ffmpeg. And I use "ps aux" found multi-process ffmpeg as Z status.
If I uninstall ffmpeg it report: OutOfRangeError (see above for traceback): PaddingFIFOQueue '_2_batch/padding_fifo_queue' is closed and has insufficient elements (requested 32, current size 0) [[Node: batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](batch/padding_fifo_queue, batch/n)]]
@tbfly I think the queue runner is not working properly. To debug, set queue=False in train1.py and then run and see what message is up.
@andabi I change the path to the currect one, but it raises：
Traceback (most recent call last):
File "train1.py", line 88, in
I checked batch_size was not changed, it was 32, and
@HudsonHuang It occurs when the number of wavfiles is smaller than the batch size.
@andabi and I check the len(wav_files) it's 0, seems it didn't get the files, I print the path in data_load.py of this line:data_load.py
and, it was
/home/lab-huang.zhongyi/workspace/deep-voice-conversion-master/datasets/timit/TIMIT/TRAIN/*/*/*.wav
and my path is :
Is it a currect path?
Thank you so much.
@andabi
2017-11-24 10:20:57.861347: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 685.56MiB 2017-11-24 10:20:57.861369: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats: Limit: 759037952 InUse: 718866432 MaxInUse: 725436928 NumAllocs: 1411 MaxAllocSize: 156712960
2017-11-24 10:20:57.861511: W tensorflow/core/common_runtime/bfc_allocator.cc:277] **************************************************************************************************** 2017-11-24 10:20:57.861549: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[32,401,2048]
It looks "GeForce GTX 750 Ti " ResourceExhaustedError when use GPU mode. 😢
Traceback (most recent call last):
File "train1.py", line 102, in
Caused by op u'Placeholder', defined at:
File "train1.py", line 102, in
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape [32,?,40] [[Node: Placeholder = Placeholderdtype=DT_FLOAT, shape=[32,?,40], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
If I set CUDA_VISIBLE_DEVICES="" force use CPU mode and set "queue = False", the error is above.
It seems CPU mode and set "queue = False": mfcc, ppg = get_batch(model.mode, model.batch_size) sess.run(train_op, feed_dict={model.x_mfcc: mfcc, model.y_ppgs: ppg})
get_batch don't get the correct tensors?
@tbfly I think your problem is similar with mine, I did‘t get currect data to feed, may I ask you what is it your path to TIMIT dataset? Was it seems like this?
@HudsonHuang your problem may be that the '.wav' at the end of the path is case sensitive and your files have .WAV instead.
data_path = '{}/timit/TIMIT/TRAIN/\*/\*/*.WAV'.format(data_path_base)
I have the same problem as @tbfly and am trying to figure it out.
@SriramS32 I think it is a os dependence or version conflict issue. I run the same code in OSX work perfect. While I run it in Linux64, it causes the issue.
@tbfly Yes, you are right. I can train on OSX with queue=True without any problems. Unfortunately, I don't have GPU support on non Linux machines. Interesting, is it expecting us to fill the placeholder even when we run the summ_op?
- summ, gs = sess.run([summ_op, global_step])
+ if queue:
+ summ, gs = sess.run([summ_op, global_step])
+ else:
+ summ, gs = sess.run([summ_op, global_step], feed_dict={model.x_mfcc: mfcc, model.y_ppgs: ppg})
@SriramS32 @andabi It seems this modify fix the error. In queue=False mode. While queue=True mode on Linux64, I need more time to figure out why not work.😄
@tbfly I meet the same problem, did you resolve it? Thank you!
Traceback (most recent call last):
File "/home/human-machine/Speech/deep-voice-conversion-master/train1.py", line 90, in
