MetricGAN
MetricGAN copied to clipboard
~loss: Nan
Hi,Jason:
When I try to train the MetricGAN(table2).py, I met two problems: (1)At every epoch both on G and D, its loss is nan , but only show a warning :
/student/home/yll/anaconda3/envs/MetricGAN/lib/python3.6/site-packages/keras/engine/training.py:973: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?
'Discrepancy between trainable weights and collected trainable'
Epoch 1/1
- 1498s - loss: nan
(2) At the fisrt epoch of traning Discriminator, it taken 593M on GPU's memory but its ultilization is 0%.
Data info: I use the same dataset from SEGAN already downsampled to 16k by the way, keras-gpu=2.1.2, tensorflow-gpu=1.10, librosa=0.5.1, python=3.6
Do you know why this happen(loss nan)? Thank you very much.
HI,
I never met this nan problem before. I'm not sure whether it is due to a different python version (mine is Python 2.7) or PESQ code. Maybe you can also try to use TargetMetric='stoi' to test the code.
Hi,Could you tell me the version of your tensorflow ,please.
Hi, it is 1.4.0