carlaILTrainer icon indicating copy to clipboard operation
carlaILTrainer copied to clipboard

OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key Network/Branch_0/Variable not found in checkpoint

Open Deepak3994 opened this issue 5 years ago • 0 comments

While running inference code python run_CIL.py, got an exception below:

2018-12-21 17:18:44.047381: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key Network/Branch_0/Variable not found in checkpoint Traceback (most recent call last): File "run_CIL.py", line 65, in agent = ImitationLearning(args.city_name, args.avoid_stopping) File "/home/deepaknayak/Documents/Reinforcement-learning/imitation-learning-inference/agents/imitation/imitation_learning.py", line 67, in init self.load_model() File "/home/deepaknayak/Documents/Reinforcement-learning/imitation-learning-inference/agents/imitation/imitation_learning.py", line 83, in load_model saver.restore(self._sess, ckpt.model_checkpoint_path) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1562, in restore err, "a Variable name or other graph key that is missing") tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key Network/Branch_0/Variable not found in checkpoint [[node save/RestoreV2 (defined at /home/deepaknayak/Documents/Reinforcement-learning/imitation-learning-inference/agents/imitation/imitation_learning.py:75) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op u'save/RestoreV2', defined at: File "run_CIL.py", line 65, in agent = ImitationLearning(args.city_name, args.avoid_stopping) File "/home/deepaknayak/Documents/Reinforcement-learning/imitation-learning-inference/agents/imitation/imitation_learning.py", line 67, in init self.load_model() File "/home/deepaknayak/Documents/Reinforcement-learning/imitation-learning-inference/agents/imitation/imitation_learning.py", line 75, in load_model saver = tf.train.Saver(variables_to_restore, max_to_keep=0) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1102, in init self.build() File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1114, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1151, in _build build_save=build_save, build_restore=build_restore) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 795, in _build_internal restore_sequentially, reshape) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 406, in _AddRestoreOps restore_sequentially) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 862, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1466, in restore_v2 shape_and_slices=shape_and_slices, dtypes=dtypes, name=name) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/deepaknayak/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key Network/Branch_0/Variable not found in checkpoint [[node save/RestoreV2 (defined at /home/deepaknayak/Documents/Reinforcement-learning/imitation-learning-inference/agents/imitation/imitation_learning.py:75) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Deepak3994 avatar Dec 21 '18 11:12 Deepak3994