tensorflow-wavenet
tensorflow-wavenet copied to clipboard
DataLossError: Unable to open table file : Failed precondition: perhaps your file is in a different file format and you need to use a different restore operator?
I modified train.py by changing SILENCE_THRESHOLD to 0.001 and modified also in generate.py. Other than that, no other changes. But I did use my own .wav files. Using a saved checkpoint model while training continues to run to test wav generation. Running with tensorflow 1.3.0, librosa 0.5.1, and python 2.7.
Here's the command that I used for training:
python train.py --data_dir=/04Training --restore_from=./logdir/train/2018-06-02T14-51-04
For wave generation:
python generate.py --wav_out_path=generated.wav --samples 16000 ./logdir/train/2018-06-02T16-51-35
Here's the error traceback:
python generate.py --wav_out_path=generated.wav --samples 16000 ./logdir/train/2018-06-02T16-51-35
/home/wilson/.local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.f
loating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Restoring model from ./logdir/train/2018-06-02T16-51-35
Traceback (most recent call last):
File "generate.py", line 289, in <module>
main()
File "generate.py", line 185, in main
saver.restore(sess, args.checkpoint)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file ./logdir/train/2018-06-02T16-51-35: Failed precondition: logdir/train/20
18-06-02T16-51-35: perhaps your file is in a different file format and you need to use a different restore operator?
[[Node: save/RestoreV2_50 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_50
/tensor_names, save/RestoreV2_50/shape_and_slices)]]
[[Node: save/RestoreV2_85/_515 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost
/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1328_save/RestoreV2_85", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/t
ask:0/gpu:0"]()]]
Caused by op u'save/RestoreV2_50', defined at:
File "generate.py", line 289, in <module>
main()
File "generate.py", line 182, in main
saver = tf.train.Saver(variables_to_restore)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
self.build()
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
filename=self._filename)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
restore_sequentially, reshape)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
dtypes=dtypes, name=name)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
DataLossError (see above for traceback): Unable to open table file ./logdir/train/2018-06-02T16-51-35: Failed precondition: logdir/train/2018-06-02T16-51
-35: perhaps your file is in a different file format and you need to use a different restore operator?
[[Node: save/RestoreV2_50 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_50
/tensor_names, save/RestoreV2_50/shape_and_slices)]]
[[Node: save/RestoreV2_85/_515 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost
/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1328_save/RestoreV2_85", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/t
ask:0/gpu:0"]()]]
Not sure if related but also received this error during training, which is why I needed to restart training with restore_from.
...
step 5816 - loss = 1.737, (1.248 sec/step)
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/playpen/wilson/tensorflow-wavenet/wavenet/audio_reader.py", line 163, in thread_main
for audio, filename, category_id in iterator:
File "/playpen/wilson/tensorflow-wavenet/wavenet/audio_reader.py", line 60, in load_generic_audio
audio, _ = librosa.load(filename, sr=sample_rate, mono=True)
File "/home/wilson/.local/lib/python2.7/site-packages/librosa/core/audio.py", line 155, in load
File "/home/wilson/.local/lib/python2.7/site-packages/librosa/core/audio.py", line 281, in resample
File "/home/wilson/.local/lib/python2.7/site-packages/resampy/core.py", line 102, in resample
File "/home/wilson/.local/lib/python2.7/site-packages/resampy/filters.py", line 167, in get_filter
NotImplementedError: Cannot load filter definition for kaiser_best
step 5817 - loss = 1.943, (1.451 sec/step)
...
I'm not sure what's going on here, but you can change the silence threshold in the command without making changes to train.py or generate.py:
python train.py --data_dir=/04Training --restore_from=./logdir/train/2018-06-02T14-51-04 --silence_threshold 0.001
I usually don't go lower than 0.1 with the silence threshold, though.