DataLossError: Unable to open table file : Failed precondition: perhaps your file is in a different file format and you need to use a different restore operator?

[jawhster]

I modified train.py by changing SILENCE_THRESHOLD to 0.001 and modified also in generate.py. Other than that, no other changes. But I did use my own .wav files. Using a saved checkpoint model while training continues to run to test wav generation. Running with tensorflow 1.3.0, librosa 0.5.1, and python 2.7.

Here's the command that I used for training: python train.py --data_dir=/04Training --restore_from=./logdir/train/2018-06-02T14-51-04

For wave generation: python generate.py --wav_out_path=generated.wav --samples 16000 ./logdir/train/2018-06-02T16-51-35

Here's the error traceback:

python generate.py --wav_out_path=generated.wav --samples 16000 ./logdir/train/2018-06-02T16-51-35
/home/wilson/.local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.f
loating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Restoring model from ./logdir/train/2018-06-02T16-51-35
Traceback (most recent call last):
  File "generate.py", line 289, in <module>
    main()
  File "generate.py", line 185, in main
    saver.restore(sess, args.checkpoint)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file ./logdir/train/2018-06-02T16-51-35: Failed precondition: logdir/train/20
18-06-02T16-51-35: perhaps your file is in a different file format and you need to use a different restore operator?
         [[Node: save/RestoreV2_50 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_50
/tensor_names, save/RestoreV2_50/shape_and_slices)]]
         [[Node: save/RestoreV2_85/_515 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost
/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1328_save/RestoreV2_85", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/t
ask:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_50', defined at:
  File "generate.py", line 289, in <module>
    main()
  File "generate.py", line 182, in main
    saver = tf.train.Saver(variables_to_restore)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/wilson/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

DataLossError (see above for traceback): Unable to open table file ./logdir/train/2018-06-02T16-51-35: Failed precondition: logdir/train/2018-06-02T16-51
-35: perhaps your file is in a different file format and you need to use a different restore operator?
         [[Node: save/RestoreV2_50 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_50
/tensor_names, save/RestoreV2_50/shape_and_slices)]]
         [[Node: save/RestoreV2_85/_515 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost
/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1328_save/RestoreV2_85", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/t
ask:0/gpu:0"]()]]

[jawhster]

Not sure if related but also received this error during training, which is why I needed to restart training with restore_from.

...
step 5816 - loss = 1.737, (1.248 sec/step)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/playpen/wilson/tensorflow-wavenet/wavenet/audio_reader.py", line 163, in thread_main
    for audio, filename, category_id in iterator:
  File "/playpen/wilson/tensorflow-wavenet/wavenet/audio_reader.py", line 60, in load_generic_audio
    audio, _ = librosa.load(filename, sr=sample_rate, mono=True)
  File "/home/wilson/.local/lib/python2.7/site-packages/librosa/core/audio.py", line 155, in load
  File "/home/wilson/.local/lib/python2.7/site-packages/librosa/core/audio.py", line 281, in resample
  File "/home/wilson/.local/lib/python2.7/site-packages/resampy/core.py", line 102, in resample
  File "/home/wilson/.local/lib/python2.7/site-packages/resampy/filters.py", line 167, in get_filter
NotImplementedError: Cannot load filter definition for kaiser_best

step 5817 - loss = 1.943, (1.451 sec/step)
...

[Benimation]

I'm not sure what's going on here, but you can change the silence threshold in the command without making changes to train.py or generate.py:

python train.py --data_dir=/04Training --restore_from=./logdir/train/2018-06-02T14-51-04 --silence_threshold 0.001

I usually don't go lower than 0.1 with the silence threshold, though.