deep-voice-conversion
deep-voice-conversion copied to clipboard
Published
20 hours ago •
andabi
andabi
Model request
Can you load a pretrained model for speech recognising, please? You know, TIMIT dataset is not very cheap and net piracy policy in several state is very strict. Thank you!
@pmsinner @akshay-ap Here's the Train1 model, with an accuracy of 70% https://drive.google.com/open?id=1ExlBIpZO0mxBhK4WEoW2WhahG1dZwdKW
Thanks!!! That will be really helpful. I went through complete source code. I have fixed all the issues. @VictoriaBentell if you want i can issue a pull request.
Can I ask for a pretrained model of the project? @VictoriaBentell @akshay-ap @pmsinner I am a student and new learner of voice conversion, it's a little hard for me to implement the project because of lacking of a good GPU in my computer now.Thank you very much!
@VictoriaBentell @andabi Hi all, I've got the 70% accuracy on a train1 architecture. and implementing net2 architecture but at every epoch loss is drastically fluctuating in range between 1 to 6. and I've used arctic dataset for traning. can anybody explain why this is happening? and It would be great if anyone provide me the trained weights of net2 architecture|?
@VictoriaBentell When I try to load your pretrained model, I always get na error "raise ValueError("No variables to save". Any idea on that?
Same for me with trying the pretrained model from https://drive.google.com/open?id=1ExlBIpZO0mxBhK4WEoW2WhahG1dZwdKW.
Traceback (most recent call last):
File "eval1.py", line 73, in <module>
eval(logdir=logdir)
File "eval1.py", line 43, in eval
predictor = OfflinePredictor(pred_conf)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorpack/predict/base.py", line 153, in __init__
config.session_init._setup_graph()
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorpack/tfutils/sessinit.py", line 114, in _setup_graph
self.saver = tf.train.Saver(var_list=dic, name=str(id(dic)))
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1239, in __init__
self.build()
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1248, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1272, in _build
raise ValueError("No variables to save")
ValueError: No variables to save
Total stack trace:
paperspace@psroeggij:~/dvc2/dvc2$ python3 eval1.py test
/home/paperspace/dvc2/dvc2/logdir/test/train1
latest checkpoint: /home/paperspace/dvc2/dvc2/logdir/test/train1/epoch_12_step_6924
[1119 02:06:43 @develop.py:96] WRN [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead!
[1119 02:06:43 @develop.py:96] WRN [Deprecated] ModelDescBase._build_graph() interface will be deprecated after 30 Mar. Use build_graph() instead!
[1119 02:06:44 @collection.py:164] These collections were modified but restored in : (tf.GraphKeys.SUMMARIES: 0->4)
[1119 02:06:44 @sessinit.py:90] WRN The following variables are in the graph, but not found in the checkpoint: net1/prenet/dense1/kernel:0, net1/prenet/dense1/bias:0, net1/prenet/dense2/kernel:0, net1/prenet/dense2/bias:0, net1/cbhg/conv1d_banks/num_1/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_1/normalize/beta:0, net1/cbhg/conv1d_banks/num_1/normalize/gamma:0, net1/cbhg/conv1d_banks/num_2/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_2/normalize/beta:0, net1/cbhg/conv1d_banks/num_2/normalize/gamma:0, net1/cbhg/conv1d_banks/num_3/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_3/normalize/beta:0, net1/cbhg/conv1d_banks/num_3/normalize/gamma:0, net1/cbhg/conv1d_banks/num_4/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_4/normalize/beta:0, net1/cbhg/conv1d_banks/num_4/normalize/gamma:0, net1/cbhg/conv1d_banks/num_5/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_5/normalize/beta:0, net1/cbhg/conv1d_banks/num_5/normalize/gamma:0, net1/cbhg/conv1d_banks/num_6/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_6/normalize/beta:0, net1/cbhg/conv1d_banks/num_6/normalize/gamma:0, net1/cbhg/conv1d_banks/num_7/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_7/normalize/beta:0, net1/cbhg/conv1d_banks/num_7/normalize/gamma:0, net1/cbhg/conv1d_banks/num_8/conv1d/conv1d/kernel:0, net1/cbhg/conv1d_banks/num_8/normalize/beta:0, net1/cbhg/conv1d_banks/num_8/normalize/gamma:0, net1/cbhg/conv1d_1/conv1d/kernel:0, net1/cbhg/normalize/beta:0, net1/cbhg/normalize/gamma:0, net1/cbhg/conv1d_2/conv1d/kernel:0, net1/cbhg/highwaynet_0/dense1/kernel:0, net1/cbhg/highwaynet_0/dense1/bias:0, net1/cbhg/highwaynet_0/dense2/kernel:0, net1/cbhg/highwaynet_0/dense2/bias:0, net1/cbhg/highwaynet_1/dense1/kernel:0, net1/cbhg/highwaynet_1/dense1/bias:0, net1/cbhg/highwaynet_1/dense2/kernel:0, net1/cbhg/highwaynet_1/dense2/bias:0, net1/cbhg/highwaynet_2/dense1/kernel:0, net1/cbhg/highwaynet_2/dense1/bias:0, net1/cbhg/highwaynet_2/dense2/kernel:0, net1/cbhg/highwaynet_2/dense2/bias:0, net1/cbhg/highwaynet_3/dense1/kernel:0, net1/cbhg/highwaynet_3/dense1/bias:0, net1/cbhg/highwaynet_3/dense2/kernel:0, net1/cbhg/highwaynet_3/dense2/bias:0, net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0, net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0, net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0, net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0, net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0, net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0, net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0, net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0, net1/dense/kernel:0, net1/dense/bias:0
[1119 02:06:44 @sessinit.py:90] WRN The following variables are in the checkpoint, but not found in the graph: beta1_power:0, beta2_power:0, global_step:0, net/net1/cbhg/conv1d_1/conv1d/kernel:0, net/net1/cbhg/conv1d_2/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_1/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_1/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_1/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_10/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_10/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_10/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_11/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_11/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_11/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_12/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_12/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_12/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_13/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_13/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_13/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_14/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_14/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_14/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_15/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_15/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_15/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_16/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_16/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_16/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_2/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_2/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_2/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_3/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_3/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_3/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_4/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_4/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_4/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_5/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_5/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_5/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_6/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_6/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_6/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_7/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_7/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_7/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_8/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_8/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_8/normalize/gamma:0, net/net1/cbhg/conv1d_banks/num_9/conv1d/conv1d/kernel:0, net/net1/cbhg/conv1d_banks/num_9/normalize/beta:0, net/net1/cbhg/conv1d_banks/num_9/normalize/gamma:0, net/net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0, net/net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0, net/net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0, net/net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0, net/net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0, net/net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0, net/net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0, net/net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0, net/net1/cbhg/highwaynet_0/dense1/bias:0, net/net1/cbhg/highwaynet_0/dense1/kernel:0, net/net1/cbhg/highwaynet_0/dense2/bias:0, net/net1/cbhg/highwaynet_0/dense2/kernel:0, net/net1/cbhg/highwaynet_1/dense1/bias:0, net/net1/cbhg/highwaynet_1/dense1/kernel:0, net/net1/cbhg/highwaynet_1/dense2/bias:0, net/net1/cbhg/highwaynet_1/dense2/kernel:0, net/net1/cbhg/highwaynet_2/dense1/bias:0, net/net1/cbhg/highwaynet_2/dense1/kernel:0, net/net1/cbhg/highwaynet_2/dense2/bias:0, net/net1/cbhg/highwaynet_2/dense2/kernel:0, net/net1/cbhg/highwaynet_3/dense1/bias:0, net/net1/cbhg/highwaynet_3/dense1/kernel:0, net/net1/cbhg/highwaynet_3/dense2/bias:0, net/net1/cbhg/highwaynet_3/dense2/kernel:0, net/net1/cbhg/normalize/beta:0, net/net1/cbhg/normalize/gamma:0, net/net1/dense/bias:0, net/net1/dense/kernel:0, net/net1/prenet/dense1/bias:0, net/net1/prenet/dense1/kernel:0, net/net1/prenet/dense2/bias:0, net/net1/prenet/dense2/kernel:0, net/net2/cbhg1/conv1d_1/conv1d/kernel:0, net/net2/cbhg1/conv1d_2/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_1/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_1/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_1/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_10/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_10/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_10/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_11/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_11/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_11/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_12/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_12/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_12/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_13/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_13/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_13/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_14/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_14/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_14/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_15/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_15/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_15/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_16/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_16/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_16/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_2/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_2/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_2/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_3/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_3/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_3/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_4/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_4/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_4/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_5/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_5/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_5/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_6/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_6/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_6/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_7/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_7/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_7/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_8/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_8/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_8/normalize/gamma:0, net/net2/cbhg1/conv1d_banks/num_9/conv1d/conv1d/kernel:0, net/net2/cbhg1/conv1d_banks/num_9/normalize/beta:0, net/net2/cbhg1/conv1d_banks/num_9/normalize/gamma:0, net/net2/cbhg1/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0, net/net2/cbhg1/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0, net/net2/cbhg1/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0, net/net2/cbhg1/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0, net/net2/cbhg1/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0, net/net2/cbhg1/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0, net/net2/cbhg1/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0, net/net2/cbhg1/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0, net/net2/cbhg1/highwaynet_0/dense1/bias:0, net/net2/cbhg1/highwaynet_0/dense1/kernel:0, net/net2/cbhg1/highwaynet_0/dense2/bias:0, net/net2/cbhg1/highwaynet_0/dense2/kernel:0, net/net2/cbhg1/highwaynet_1/dense1/bias:0, net/net2/cbhg1/highwaynet_1/dense1/kernel:0, net/net2/cbhg1/highwaynet_1/dense2/bias:0, net/net2/cbhg1/highwaynet_1/dense2/kernel:0, net/net2/cbhg1/highwaynet_2/dense1/bias:0, net/net2/cbhg1/highwaynet_2/dense1/kernel:0, net/net2/cbhg1/highwaynet_2/dense2/bias:0, net/net2/cbhg1/highwaynet_2/dense2/kernel:0, net/net2/cbhg1/highwaynet_3/dense1/bias:0, net/net2/cbhg1/highwaynet_3/dense1/kernel:0, net/net2/cbhg1/highwaynet_3/dense2/bias:0, net/net2/cbhg1/highwaynet_3/dense2/kernel:0, net/net2/cbhg1/highwaynet_4/dense1/bias:0, net/net2/cbhg1/highwaynet_4/dense1/kernel:0, net/net2/cbhg1/highwaynet_4/dense2/bias:0, net/net2/cbhg1/highwaynet_4/dense2/kernel:0, net/net2/cbhg1/highwaynet_5/dense1/bias:0, net/net2/cbhg1/highwaynet_5/dense1/kernel:0, net/net2/cbhg1/highwaynet_5/dense2/bias:0, net/net2/cbhg1/highwaynet_5/dense2/kernel:0, net/net2/cbhg1/highwaynet_6/dense1/bias:0, net/net2/cbhg1/highwaynet_6/dense1/kernel:0, net/net2/cbhg1/highwaynet_6/dense2/bias:0, net/net2/cbhg1/highwaynet_6/dense2/kernel:0, net/net2/cbhg1/highwaynet_7/dense1/bias:0, net/net2/cbhg1/highwaynet_7/dense1/kernel:0, net/net2/cbhg1/highwaynet_7/dense2/bias:0, net/net2/cbhg1/highwaynet_7/dense2/kernel:0, net/net2/cbhg1/normalize/beta:0, net/net2/cbhg1/normalize/gamma:0, net/net2/cbhg2/conv1d_1/conv1d/kernel:0, net/net2/cbhg2/conv1d_2/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_1/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_1/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_1/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_10/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_10/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_10/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_11/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_11/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_11/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_12/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_12/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_12/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_13/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_13/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_13/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_14/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_14/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_14/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_15/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_15/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_15/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_16/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_16/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_16/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_2/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_2/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_2/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_3/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_3/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_3/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_4/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_4/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_4/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_5/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_5/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_5/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_6/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_6/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_6/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_7/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_7/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_7/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_8/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_8/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_8/normalize/gamma:0, net/net2/cbhg2/conv1d_banks/num_9/conv1d/conv1d/kernel:0, net/net2/cbhg2/conv1d_banks/num_9/normalize/beta:0, net/net2/cbhg2/conv1d_banks/num_9/normalize/gamma:0, net/net2/cbhg2/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0, net/net2/cbhg2/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0, net/net2/cbhg2/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0, net/net2/cbhg2/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0, net/net2/cbhg2/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0, net/net2/cbhg2/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0, net/net2/cbhg2/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0, net/net2/cbhg2/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0, net/net2/cbhg2/highwaynet_0/dense1/bias:0, net/net2/cbhg2/highwaynet_0/dense1/kernel:0, net/net2/cbhg2/highwaynet_0/dense2/bias:0, net/net2/cbhg2/highwaynet_0/dense2/kernel:0, net/net2/cbhg2/highwaynet_1/dense1/bias:0, net/net2/cbhg2/highwaynet_1/dense1/kernel:0, net/net2/cbhg2/highwaynet_1/dense2/bias:0, net/net2/cbhg2/highwaynet_1/dense2/kernel:0, net/net2/cbhg2/highwaynet_2/dense1/bias:0, net/net2/cbhg2/highwaynet_2/dense1/kernel:0, net/net2/cbhg2/highwaynet_2/dense2/bias:0, net/net2/cbhg2/highwaynet_2/dense2/kernel:0, net/net2/cbhg2/highwaynet_3/dense1/bias:0, net/net2/cbhg2/highwaynet_3/dense1/kernel:0, net/net2/cbhg2/highwaynet_3/dense2/bias:0, net/net2/cbhg2/highwaynet_3/dense2/kernel:0, net/net2/cbhg2/highwaynet_4/dense1/bias:0, net/net2/cbhg2/highwaynet_4/dense1/kernel:0, net/net2/cbhg2/highwaynet_4/dense2/bias:0, net/net2/cbhg2/highwaynet_4/dense2/kernel:0, net/net2/cbhg2/highwaynet_5/dense1/bias:0, net/net2/cbhg2/highwaynet_5/dense1/kernel:0, net/net2/cbhg2/highwaynet_5/dense2/bias:0, net/net2/cbhg2/highwaynet_5/dense2/kernel:0, net/net2/cbhg2/highwaynet_6/dense1/bias:0, net/net2/cbhg2/highwaynet_6/dense1/kernel:0, net/net2/cbhg2/highwaynet_6/dense2/bias:0, net/net2/cbhg2/highwaynet_6/dense2/kernel:0, net/net2/cbhg2/highwaynet_7/dense1/bias:0, net/net2/cbhg2/highwaynet_7/dense1/kernel:0, net/net2/cbhg2/highwaynet_7/dense2/bias:0, net/net2/cbhg2/highwaynet_7/dense2/kernel:0, net/net2/cbhg2/normalize/beta:0, net/net2/cbhg2/normalize/gamma:0, net/net2/dense/bias:0, net/net2/dense/kernel:0, net/net2/dense_1/bias:0, net/net2/dense_1/kernel:0, net/net2/dense_2/bias:0, net/net2/dense_2/kernel:0, net/net2/prenet/dense1/bias:0, net/net2/prenet/dense1/kernel:0, net/net2/prenet/dense2/bias:0, net/net2/prenet/dense2/kernel:0
Traceback (most recent call last):
File "eval1.py", line 73, in <module>
eval(logdir=logdir)
File "eval1.py", line 43, in eval
predictor = OfflinePredictor(pred_conf)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorpack/predict/base.py", line 153, in __init__
config.session_init._setup_graph()
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorpack/tfutils/sessinit.py", line 114, in _setup_graph
self.saver = tf.train.Saver(var_list=dic, name=str(id(dic)))
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1239, in __init__
self.build()
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1248, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1272, in _build
raise ValueError("No variables to save")
ValueError: No variables to save
Also a bit confused with that model where the size and filename comes from. Keeping all the default hyperparams, train1.py is generating models like the following for me:
I trained the model in the Google Colab for 6 hours and got 93% accuracy...Here is the model for train1.py code. https://drive.google.com/file/d/1yC3G3V03X3s8mKJ1J6bMkOqDT8r-TBb8/view?usp=sharing Use Tensorboard to view the confusion matrix for this model output. sudo pip install tensorflow cd train1 tensorboard --logdir . open in chrome : localhost:6006
LJ Speech dataset link: https://keithito.com/LJ-Speech-Dataset/
Thanks. Where does the 93% accuracy figure come from? In the readme it states only "over 70% test accuracy" for the first stage of training. Same with https://github.com/andabi/deep-voice-conversion/issues/12#issuecomment-373473205 (70% accuracy).
That depends upon how many epochs you train for. With Google Colab having the best Nvidia K80 Tesla GPu (4 of them) you can train the model much faster. Try to plot the accuracy curve from the model files I provided earlier in the Tensorboard for proof. It took 54 epochs to reach that accuracy.
@gouravsb17 I'm trying to use your model, but hitting upon this error when starting train2 (which doesn't happen if I use a model that was trained on my PC):
Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint
Do you know what could be the case?
@DefiantCatgirl Can u further elucidate what are u exactly trying to do? I provided the train1 model. Once u load it and run followed by training on 2 it should work.
@rahulkhairnarr @DefiantCatgirl Here is the link to google colab notebook I used. [https://colab.research.google.com/drive/1OoEZ2ouByr857UX-KAGT1hkXcUirJwdP] You can download the files and codes I provided earlier in the google drive link and put them respectively in the content folder of the colab to run the notebook.
@gouravsb17Inno This is the error log I'm getting when I'm trying to run train2 on my machine using the models you provided.
Error log
PS D:\Downloads\Software\Audio\deep-voice-conversion-master> python3 .\train2.py -case gouravsb
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
case: gouravsb, logdir1: cases/gouravsb/train1, logdir2: cases/gouravsb/train2
[32m[0516 14:06:37 @logger.py:73][0m Argv: .\train2.py -case gouravsb
[32m[0516 14:06:37 @parallel.py:176][0m [5m[31mWRN[0m MultiProcessPrefetchData does support windows. However, windows requires more strict picklability on processes, which may lead of failure on some of the code.
[32m[0516 14:06:37 @parallel.py:186][0m [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[32m[0516 14:06:37 @sesscreate.py:38][0m [5m[31mWRN[0m User-provided custom session config may not work due to TF bugs. See https://github.com/tensorpack/tensorpack/issues/497 for workarounds.
[32m[0516 14:06:37 @develop.py:96][0m [5m[31mWRN[0m [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead!
[32m[0516 14:06:37 @input_source.py:220][0m Setting up the queue 'QueueInput/input_queue' for CPU prefetching ...
WARNING:tensorflow:From E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[32m[0516 14:06:37 @trainers.py:52][0m Building graph for a single training tower ...
[32m[0516 14:06:37 @develop.py:96][0m [5m[31mWRN[0m [Deprecated] ModelDescBase._build_graph() interface will be deprecated after 30 Mar. Use build_graph() instead!
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\modules.py:276: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\modules.py:278: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
WARNING:tensorflow:From E:\Python\Python36-64\lib\site-packages\tensorflow\python\keras\layers\core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\modules.py:169: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv1d instead.
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\modules.py:323: max_pooling1d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling1d instead.
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\modules.py:217: GRUCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.GRUCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\modules.py:222: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From E:\Python\Python36-64\lib\site-packages\tensorflow\python\ops\rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From D:\Downloads\Software\Audio\deep-voice-conversion-master\models.py:68: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
[32m[0516 14:06:43 @develop.py:96][0m [5m[31mWRN[0m [Deprecated] get_cost() and self.cost will be deprecated after 30 Mar. Return the cost tensor directly in build_graph() instead!
[32m[0516 14:06:43 @develop.py:96][0m [5m[31mWRN[0m [Deprecated] ModelDescBase._get_optimizer() interface will be deprecated after 30 Mar. Use optimizer() instead!
[32m[0516 14:07:03 @model_utils.py:64][0m [36mTrainable Variables:
[0mname shape dim
--------------------------------------------------------------------- -------------- ------
net1/prenet/dense1/kernel:0 [60, 128] 7680
net1/prenet/dense1/bias:0 [128] 128
net1/prenet/dense2/kernel:0 [128, 64] 8192
net1/prenet/dense2/bias:0 [64] 64
net1/cbhg/conv1d_banks/num_1/conv1d/conv1d/kernel:0 [1, 64, 64] 4096
net1/cbhg/conv1d_banks/num_1/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_1/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_2/conv1d/conv1d/kernel:0 [2, 64, 64] 8192
net1/cbhg/conv1d_banks/num_2/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_2/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_3/conv1d/conv1d/kernel:0 [3, 64, 64] 12288
net1/cbhg/conv1d_banks/num_3/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_3/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_4/conv1d/conv1d/kernel:0 [4, 64, 64] 16384
net1/cbhg/conv1d_banks/num_4/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_4/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_5/conv1d/conv1d/kernel:0 [5, 64, 64] 20480
net1/cbhg/conv1d_banks/num_5/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_5/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_6/conv1d/conv1d/kernel:0 [6, 64, 64] 24576
net1/cbhg/conv1d_banks/num_6/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_6/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_7/conv1d/conv1d/kernel:0 [7, 64, 64] 28672
net1/cbhg/conv1d_banks/num_7/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_7/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_banks/num_8/conv1d/conv1d/kernel:0 [8, 64, 64] 32768
net1/cbhg/conv1d_banks/num_8/normalize/beta:0 [64] 64
net1/cbhg/conv1d_banks/num_8/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_1/conv1d/kernel:0 [3, 512, 64] 98304
net1/cbhg/normalize/beta:0 [64] 64
net1/cbhg/normalize/gamma:0 [64] 64
net1/cbhg/conv1d_2/conv1d/kernel:0 [3, 64, 64] 12288
net1/cbhg/highwaynet_0/dense1/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_0/dense1/bias:0 [64] 64
net1/cbhg/highwaynet_0/dense2/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_0/dense2/bias:0 [64] 64
net1/cbhg/highwaynet_1/dense1/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_1/dense1/bias:0 [64] 64
net1/cbhg/highwaynet_1/dense2/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_1/dense2/bias:0 [64] 64
net1/cbhg/highwaynet_2/dense1/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_2/dense1/bias:0 [64] 64
net1/cbhg/highwaynet_2/dense2/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_2/dense2/bias:0 [64] 64
net1/cbhg/highwaynet_3/dense1/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_3/dense1/bias:0 [64] 64
net1/cbhg/highwaynet_3/dense2/kernel:0 [64, 64] 4096
net1/cbhg/highwaynet_3/dense2/bias:0 [64] 64
net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0 [128, 128] 16384
net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0 [128] 128
net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0 [128, 64] 8192
net1/cbhg/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0 [64] 64
net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0 [128, 128] 16384
net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0 [128] 128
net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0 [128, 64] 8192
net1/cbhg/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0 [64] 64
net1/dense/kernel:0 [128, 61] 7808
net1/dense/bias:0 [61] 61
net2/prenet/dense1/kernel:0 [61, 256] 15616
net2/prenet/dense1/bias:0 [256] 256
net2/prenet/dense2/kernel:0 [256, 128] 32768
net2/prenet/dense2/bias:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_1/conv1d/conv1d/kernel:0 [1, 128, 128] 16384
net2/cbhg_mel/conv1d_banks/num_1/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_1/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_2/conv1d/conv1d/kernel:0 [2, 128, 128] 32768
net2/cbhg_mel/conv1d_banks/num_2/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_2/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_3/conv1d/conv1d/kernel:0 [3, 128, 128] 49152
net2/cbhg_mel/conv1d_banks/num_3/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_3/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_4/conv1d/conv1d/kernel:0 [4, 128, 128] 65536
net2/cbhg_mel/conv1d_banks/num_4/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_4/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_5/conv1d/conv1d/kernel:0 [5, 128, 128] 81920
net2/cbhg_mel/conv1d_banks/num_5/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_5/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_6/conv1d/conv1d/kernel:0 [6, 128, 128] 98304
net2/cbhg_mel/conv1d_banks/num_6/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_6/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_7/conv1d/conv1d/kernel:0 [7, 128, 128] 114688
net2/cbhg_mel/conv1d_banks/num_7/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_7/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_8/conv1d/conv1d/kernel:0 [8, 128, 128] 131072
net2/cbhg_mel/conv1d_banks/num_8/normalize/beta:0 [128] 128
net2/cbhg_mel/conv1d_banks/num_8/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_1/conv1d/kernel:0 [3, 1024, 128] 393216
net2/cbhg_mel/normalize/beta:0 [128] 128
net2/cbhg_mel/normalize/gamma:0 [128] 128
net2/cbhg_mel/conv1d_2/conv1d/kernel:0 [3, 128, 128] 49152
net2/cbhg_mel/highwaynet_0/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_0/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_0/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_0/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_1/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_1/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_1/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_1/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_2/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_2/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_2/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_2/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_3/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_3/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_3/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_3/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_4/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_4/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_4/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_4/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_5/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_5/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_5/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_5/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_6/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_6/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_6/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_6/dense2/bias:0 [128] 128
net2/cbhg_mel/highwaynet_7/dense1/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_7/dense1/bias:0 [128] 128
net2/cbhg_mel/highwaynet_7/dense2/kernel:0 [128, 128] 16384
net2/cbhg_mel/highwaynet_7/dense2/bias:0 [128] 128
net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0 [256, 256] 65536
net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0 [256] 256
net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0 [256, 128] 32768
net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0 [128] 128
net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0 [256, 256] 65536
net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0 [256] 256
net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0 [256, 128] 32768
net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0 [128] 128
net2/pred_mel/kernel:0 [256, 90] 23040
net2/pred_mel/bias:0 [90] 90
net2/dense/kernel:0 [90, 128] 11520
net2/dense/bias:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_1/conv1d/conv1d/kernel:0 [1, 128, 128] 16384
net2/cbhg_linear/conv1d_banks/num_1/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_1/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_2/conv1d/conv1d/kernel:0 [2, 128, 128] 32768
net2/cbhg_linear/conv1d_banks/num_2/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_2/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_3/conv1d/conv1d/kernel:0 [3, 128, 128] 49152
net2/cbhg_linear/conv1d_banks/num_3/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_3/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_4/conv1d/conv1d/kernel:0 [4, 128, 128] 65536
net2/cbhg_linear/conv1d_banks/num_4/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_4/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_5/conv1d/conv1d/kernel:0 [5, 128, 128] 81920
net2/cbhg_linear/conv1d_banks/num_5/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_5/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_6/conv1d/conv1d/kernel:0 [6, 128, 128] 98304
net2/cbhg_linear/conv1d_banks/num_6/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_6/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_7/conv1d/conv1d/kernel:0 [7, 128, 128] 114688
net2/cbhg_linear/conv1d_banks/num_7/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_7/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_8/conv1d/conv1d/kernel:0 [8, 128, 128] 131072
net2/cbhg_linear/conv1d_banks/num_8/normalize/beta:0 [128] 128
net2/cbhg_linear/conv1d_banks/num_8/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_1/conv1d/kernel:0 [3, 1024, 128] 393216
net2/cbhg_linear/normalize/beta:0 [128] 128
net2/cbhg_linear/normalize/gamma:0 [128] 128
net2/cbhg_linear/conv1d_2/conv1d/kernel:0 [3, 128, 128] 49152
net2/cbhg_linear/highwaynet_0/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_0/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_0/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_0/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_1/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_1/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_1/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_1/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_2/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_2/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_2/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_2/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_3/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_3/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_3/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_3/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_4/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_4/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_4/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_4/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_5/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_5/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_5/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_5/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_6/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_6/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_6/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_6/dense2/bias:0 [128] 128
net2/cbhg_linear/highwaynet_7/dense1/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_7/dense1/bias:0 [128] 128
net2/cbhg_linear/highwaynet_7/dense2/kernel:0 [128, 128] 16384
net2/cbhg_linear/highwaynet_7/dense2/bias:0 [128] 128
net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0 [256, 256] 65536
net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0 [256] 256
net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0 [256, 128] 32768
net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0 [128] 128
net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0 [256, 256] 65536
net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0 [256] 256
net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0 [256, 128] 32768
net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0 [128] 128
net2/pred_spec/kernel:0 [256, 569] 145664
net2/pred_spec/bias:0 [569] 569[36m
Total #vars=204, #params=3587856, size=13.69MB[0m
[32m[0516 14:07:03 @base.py:209][0m Setup callbacks graph ...
[32m[0516 14:07:04 @summary.py:38][0m Maintain moving average summary of 0 tensors in collection MOVING_SUMMARY_OPS.
[32m[0516 14:07:04 @summary.py:75][0m Summarizing collection 'summaries' of size 2.
[32m[0516 14:07:11 @sessinit.py:137][0m Variable global_step:0 in the graph will not be loaded from the checkpoint!
[32m[0516 14:07:11 @sessinit.py:90][0m [5m[31mWRN[0m The following variables are in the graph, but not found in the checkpoint: net2/prenet/dense1/kernel:0, net2/prenet/dense1/bias:0, net2/prenet/dense2/kernel:0, net2/prenet/dense2/bias:0, net2/cbhg_mel/conv1d_banks/num_1/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_1/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_1/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_2/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_2/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_2/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_3/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_3/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_3/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_4/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_4/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_4/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_5/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_5/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_5/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_6/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_6/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_6/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_7/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_7/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_7/normalize/gamma:0, net2/cbhg_mel/conv1d_banks/num_8/conv1d/conv1d/kernel:0, net2/cbhg_mel/conv1d_banks/num_8/normalize/beta:0, net2/cbhg_mel/conv1d_banks/num_8/normalize/gamma:0, net2/cbhg_mel/conv1d_1/conv1d/kernel:0, net2/cbhg_mel/normalize/beta:0, net2/cbhg_mel/normalize/gamma:0, net2/cbhg_mel/conv1d_2/conv1d/kernel:0, net2/cbhg_mel/highwaynet_0/dense1/kernel:0, net2/cbhg_mel/highwaynet_0/dense1/bias:0, net2/cbhg_mel/highwaynet_0/dense2/kernel:0, net2/cbhg_mel/highwaynet_0/dense2/bias:0, net2/cbhg_mel/highwaynet_1/dense1/kernel:0, net2/cbhg_mel/highwaynet_1/dense1/bias:0, net2/cbhg_mel/highwaynet_1/dense2/kernel:0, net2/cbhg_mel/highwaynet_1/dense2/bias:0, net2/cbhg_mel/highwaynet_2/dense1/kernel:0, net2/cbhg_mel/highwaynet_2/dense1/bias:0, net2/cbhg_mel/highwaynet_2/dense2/kernel:0, net2/cbhg_mel/highwaynet_2/dense2/bias:0, net2/cbhg_mel/highwaynet_3/dense1/kernel:0, net2/cbhg_mel/highwaynet_3/dense1/bias:0, net2/cbhg_mel/highwaynet_3/dense2/kernel:0, net2/cbhg_mel/highwaynet_3/dense2/bias:0, net2/cbhg_mel/highwaynet_4/dense1/kernel:0, net2/cbhg_mel/highwaynet_4/dense1/bias:0, net2/cbhg_mel/highwaynet_4/dense2/kernel:0, net2/cbhg_mel/highwaynet_4/dense2/bias:0, net2/cbhg_mel/highwaynet_5/dense1/kernel:0, net2/cbhg_mel/highwaynet_5/dense1/bias:0, net2/cbhg_mel/highwaynet_5/dense2/kernel:0, net2/cbhg_mel/highwaynet_5/dense2/bias:0, net2/cbhg_mel/highwaynet_6/dense1/kernel:0, net2/cbhg_mel/highwaynet_6/dense1/bias:0, net2/cbhg_mel/highwaynet_6/dense2/kernel:0, net2/cbhg_mel/highwaynet_6/dense2/bias:0, net2/cbhg_mel/highwaynet_7/dense1/kernel:0, net2/cbhg_mel/highwaynet_7/dense1/bias:0, net2/cbhg_mel/highwaynet_7/dense2/kernel:0, net2/cbhg_mel/highwaynet_7/dense2/bias:0, net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0, net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0, net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0, net2/cbhg_mel/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0, net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0, net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0, net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0, net2/cbhg_mel/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0, net2/pred_mel/kernel:0, net2/pred_mel/bias:0, net2/dense/kernel:0, net2/dense/bias:0, net2/cbhg_linear/conv1d_banks/num_1/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_1/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_1/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_2/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_2/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_2/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_3/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_3/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_3/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_4/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_4/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_4/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_5/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_5/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_5/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_6/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_6/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_6/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_7/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_7/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_7/normalize/gamma:0, net2/cbhg_linear/conv1d_banks/num_8/conv1d/conv1d/kernel:0, net2/cbhg_linear/conv1d_banks/num_8/normalize/beta:0, net2/cbhg_linear/conv1d_banks/num_8/normalize/gamma:0, net2/cbhg_linear/conv1d_1/conv1d/kernel:0, net2/cbhg_linear/normalize/beta:0, net2/cbhg_linear/normalize/gamma:0, net2/cbhg_linear/conv1d_2/conv1d/kernel:0, net2/cbhg_linear/highwaynet_0/dense1/kernel:0, net2/cbhg_linear/highwaynet_0/dense1/bias:0, net2/cbhg_linear/highwaynet_0/dense2/kernel:0, net2/cbhg_linear/highwaynet_0/dense2/bias:0, net2/cbhg_linear/highwaynet_1/dense1/kernel:0, net2/cbhg_linear/highwaynet_1/dense1/bias:0, net2/cbhg_linear/highwaynet_1/dense2/kernel:0, net2/cbhg_linear/highwaynet_1/dense2/bias:0, net2/cbhg_linear/highwaynet_2/dense1/kernel:0, net2/cbhg_linear/highwaynet_2/dense1/bias:0, net2/cbhg_linear/highwaynet_2/dense2/kernel:0, net2/cbhg_linear/highwaynet_2/dense2/bias:0, net2/cbhg_linear/highwaynet_3/dense1/kernel:0, net2/cbhg_linear/highwaynet_3/dense1/bias:0, net2/cbhg_linear/highwaynet_3/dense2/kernel:0, net2/cbhg_linear/highwaynet_3/dense2/bias:0, net2/cbhg_linear/highwaynet_4/dense1/kernel:0, net2/cbhg_linear/highwaynet_4/dense1/bias:0, net2/cbhg_linear/highwaynet_4/dense2/kernel:0, net2/cbhg_linear/highwaynet_4/dense2/bias:0, net2/cbhg_linear/highwaynet_5/dense1/kernel:0, net2/cbhg_linear/highwaynet_5/dense1/bias:0, net2/cbhg_linear/highwaynet_5/dense2/kernel:0, net2/cbhg_linear/highwaynet_5/dense2/bias:0, net2/cbhg_linear/highwaynet_6/dense1/kernel:0, net2/cbhg_linear/highwaynet_6/dense1/bias:0, net2/cbhg_linear/highwaynet_6/dense2/kernel:0, net2/cbhg_linear/highwaynet_6/dense2/bias:0, net2/cbhg_linear/highwaynet_7/dense1/kernel:0, net2/cbhg_linear/highwaynet_7/dense1/bias:0, net2/cbhg_linear/highwaynet_7/dense2/kernel:0, net2/cbhg_linear/highwaynet_7/dense2/bias:0, net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/gates/kernel:0, net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/gates/bias:0, net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/candidate/kernel:0, net2/cbhg_linear/gru/bidirectional_rnn/fw/gru_cell/candidate/bias:0, net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/gates/kernel:0, net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/gates/bias:0, net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/candidate/kernel:0, net2/cbhg_linear/gru/bidirectional_rnn/bw/gru_cell/candidate/bias:0, net2/pred_spec/kernel:0, net2/pred_spec/bias:0, beta1_power:0, beta2_power:0
[32m[0516 14:07:11 @sessinit.py:90][0m [5m[31mWRN[0m The following variables are in the checkpoint, but not found in the graph: global_step:0
[32m[0516 14:07:11 @base.py:227][0m Creating the session ...
2019-05-16 14:07:11.487655: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-05-16 14:07:12.580095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 970M major: 5 minor: 2 memoryClockRate(GHz): 1.038
pciBusID: 0000:01:00.0
totalMemory: 3.00GiB freeMemory: 2.47GiB
2019-05-16 14:07:12.596633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-16 14:07:15.679717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-16 14:07:15.686346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-05-16 14:07:15.695449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-05-16 14:07:15.710179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2159 MB memory) -> physical GPU (device: 0, name: GeForce GTX 970M, pci bus id: 0000:01:00.0, compute capability: 5.2)
[32m[0516 14:07:21 @base.py:233][0m Initializing the session ...
[32m[0516 14:07:21 @sessinit.py:117][0m Restoring checkpoint from cases/gouravsb/train1\model-5300 ...
WARNING:tensorflow:From E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [60,128] rhs shape= [40,128]
[[{{node 2105436627664/Assign_59}}]]
[[{{node 2105436627664/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 1276, in restore
{self.saver_def.filename_tensor_name: save_path})
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [60,128] rhs shape= [40,128]
[[node 2105436627664/Assign_59 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
[[node 2105436627664/RestoreV2 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
Caused by op '2105436627664/Assign_59', defined at:
File ".\train2.py", line 95, in <module>
train(args, logdir1=logdir_train1, logdir2=logdir_train2)
File ".\train2.py", line 64, in train
launch_train_with_config(train_conf, trainer=trainer)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\interface.py", line 97, in launch_train_with_config
extra_callbacks=config.extra_callbacks)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 341, in train_with_defaults
steps_per_epoch, starting_epoch, max_epoch)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 312, in train
self.initialize(session_creator, session_init)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\utils\argtools.py", line 176, in wrapper
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\tower.py", line 144, in initialize
super(TowerTrainer, self).initialize(session_creator, session_init)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\utils\argtools.py", line 176, in wrapper
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 225, in initialize
session_init._setup_graph()
File "E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py", line 239, in _setup_graph
i._setup_graph()
File "E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py", line 114, in _setup_graph
self.saver = tf.train.Saver(var_list=dic, name=str(id(dic)))
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 832, in __init__
self.build()
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 881, in _build
build_save=build_save, build_restore=build_restore)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 513, in _build_internal
restore_sequentially, reshape)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 354, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saving\saveable_object_util.py", line 73, in restore
self.op.get_shape().is_fully_defined())
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\ops\state_ops.py", line 223, in assign
validate_shape=validate_shape)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 68, in assign
use_locking=use_locking, name=name)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [60,128] rhs shape= [40,128]
[[node 2105436627664/Assign_59 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
[[node 2105436627664/RestoreV2 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".\train2.py", line 95, in <module>
train(args, logdir1=logdir_train1, logdir2=logdir_train2)
File ".\train2.py", line 64, in train
launch_train_with_config(train_conf, trainer=trainer)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\interface.py", line 97, in launch_train_with_config
extra_callbacks=config.extra_callbacks)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 341, in train_with_defaults
steps_per_epoch, starting_epoch, max_epoch)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 312, in train
self.initialize(session_creator, session_init)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\utils\argtools.py", line 176, in wrapper
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\tower.py", line 144, in initialize
super(TowerTrainer, self).initialize(session_creator, session_init)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\utils\argtools.py", line 176, in wrapper
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 234, in initialize
session_init._run_init(self.sess)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py", line 243, in _run_init
i._run_init(sess)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py", line 118, in _run_init
self.saver.restore(sess, self.path)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 1312, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [60,128] rhs shape= [40,128]
[[node 2105436627664/Assign_59 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
[[node 2105436627664/RestoreV2 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
Caused by op '2105436627664/Assign_59', defined at:
File ".\train2.py", line 95, in <module>
train(args, logdir1=logdir_train1, logdir2=logdir_train2)
File ".\train2.py", line 64, in train
launch_train_with_config(train_conf, trainer=trainer)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\interface.py", line 97, in launch_train_with_config
extra_callbacks=config.extra_callbacks)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 341, in train_with_defaults
steps_per_epoch, starting_epoch, max_epoch)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 312, in train
self.initialize(session_creator, session_init)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\utils\argtools.py", line 176, in wrapper
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\tower.py", line 144, in initialize
super(TowerTrainer, self).initialize(session_creator, session_init)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\utils\argtools.py", line 176, in wrapper
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorpack\train\base.py", line 225, in initialize
session_init._setup_graph()
File "E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py", line 239, in _setup_graph
i._setup_graph()
File "E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py", line 114, in _setup_graph
self.saver = tf.train.Saver(var_list=dic, name=str(id(dic)))
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 832, in __init__
self.build()
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 881, in _build
build_save=build_save, build_restore=build_restore)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 513, in _build_internal
restore_sequentially, reshape)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saver.py", line 354, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\training\saving\saveable_object_util.py", line 73, in restore
self.op.get_shape().is_fully_defined())
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\ops\state_ops.py", line 223, in assign
validate_shape=validate_shape)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 68, in assign
use_locking=use_locking, name=name)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "E:\Python\Python36-64\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [60,128] rhs shape= [40,128]
[[node 2105436627664/Assign_59 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
[[node 2105436627664/RestoreV2 (defined at E:\Python\Python36-64\lib\site-packages\tensorpack\tfutils\sessinit.py:114) ]]
Perhaps it's an underlying issue and TensorFlow doesn't make it easy to use models trained on a different GPU (train1 results from my machine work fine with train2), I have genuinely no idea even after googling around. I will try using this on colab and see if it works there. In any case, thanks a lot for the models.
@DefiantCatgirl InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [60,128] rhs shape= [40,128]
check ur default.yaml file. It should look as below:
default: sr: 16000 frame_shift: 0.005 frame_length: 0.025 win_length: 400 hop_length: 80 n_fft: 512 preemphasis: 0.97 n_mfcc: 40 n_iter: 60 # Number of inversion iterations n_mels: 80 duration: 2 max_db: 35 min_db: -55
# model
hidden_units: 256 # alias: E
num_banks: 16
num_highway_blocks: 4
norm_type: 'ins' # a normalizer function. value: bn, ln, ins, or None
t: 1.0 # temperature
dropout_rate: 0.2
# train
batch_size: 32
logdir_path: '/data/private/vc/logdir'
train1: # path ##data_path: '/data/private/vc/datasets/timit/TIMIT/TRAIN///.wav' data_path: 'datasets/timit/TIMIT/TRAIN//*.wav' # model hidden_units: 128 # alias: E num_banks: 8 num_highway_blocks: 4 norm_type: 'ins' # a normalizer function. value: bn, ln, ins, or None t: 1.0 # temperature dropout_rate: 0.2
# train
batch_size: 32
lr: 0.0003
num_epochs: 1000
steps_per_epoch: 100
save_per_epoch: 2
num_gpu: 2
train2: # path ##data_path: '/data/private/vc/datasets/arctic/slt/.wav' data_path: 'datasets/arctic/slt/.wav'
# model
hidden_units: 256 # alias: E
num_banks: 8
num_highway_blocks: 8
norm_type: 'ins' # a normalizer function. value: bn, ln, ins, or None
t: 1.0 # temperature
dropout_rate: 0.2
# train
batch_size: 32
lr: 0.0003
lr_cyclic_margin: 0.
lr_cyclic_steps: 5000
clip_value_max: 3.
clip_value_min: -3.
clip_norm: 10
num_epochs: 1000
steps_per_epoch: 100
save_per_epoch: 50
test_per_epoch: 1
num_gpu: 4
test1: # path ##data_path: '/data/private/vc/datasets/timit/TIMIT/TEST///.wav' data_path: 'datasets/timit/TIMIT/TEST//*.wav'
# test
batch_size: 32
test2: # path ##data_path: '/data/private/vc/datasets/arctic/slt/.wav' data_path: 'datasets/arctic/slt/.wav'
# test
batch_size: 32
convert: # path ##data_path: '/data/private/vc/datasets/arctic/bdl/*.wav' data_path: 'datasets/arctic/bdl/f2rporam.wav'
# convert
one_full_wav: True
batch_size: 1
emphasis_magnitude: 1.2
That appears to have worked, at least the training successfully started, thank you!
Did you guys ever run into raise NotImplementedError() NotImplementedError?
I am also trying to run python train2.py and am using @gouravsb17Inno model as well as using the above default.yaml configs. Thanks!
Also I used a hack fix presented by someone else by using tensorpack 0.9.0.1 but I am stuck on epoch 1 forever
@gouravsb17 Can You please provide me the train2 pretrained model weights?? Please Help
