Error when trying to restore all layers and training the whole model

[alex-razor]

This is in the args.py:

# choise 3: restore all layers
restore_include = None
restore_exclude = None
# Choose the parts you want to finetune. List form.
# Set to None to train the whole model.
update_part = None

And this is the error:

raceback (most recent call last):
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1292, in _do_call
    return fn(*args)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1277, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1367, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1,1,512,18] rhs shape= [1,1,512,255]
	 [[{{node save/Assign_291}} = Assign[T=DT_FLOAT, _class=["loc:@yolov3/yolov3_head/Conv_14/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/Conv_14/weights, save/RestoreV2/_583)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1538, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1308, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1,1,512,18] rhs shape= [1,1,512,255]
	 [[{{node save/Assign_291}} = Assign[T=DT_FLOAT, _class=["loc:@yolov3/yolov3_head/Conv_14/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/Conv_14/weights, save/RestoreV2/_583)]]

Caused by op 'save/Assign_291', defined at:
  File "/hdd/YOLOv3_TensorFlow/train.py", line 81, in <module>
    saver_to_restore = tf.train.Saver(var_list=tf.contrib.framework.get_variables_to_restore(include=args.restore_include, exclude=args.restore_exclude))
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1094, in __init__
    self.build()
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1106, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1143, in _build
    build_save=build_save, build_restore=build_restore)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 787, in _build_internal
    restore_sequentially, reshape)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 119, in restore
    self.op.get_shape().is_fully_defined())
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
    validate_shape=validate_shape)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
    use_locking=use_locking, name=name)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3272, in create_op
    op_def=op_def)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1,1,512,18] rhs shape= [1,1,512,255]
	 [[{{node save/Assign_291}} = Assign[T=DT_FLOAT, _class=["loc:@yolov3/yolov3_head/Conv_14/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/Conv_14/weights, save/RestoreV2/_583)]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/hdd/YOLOv3_TensorFlow/train.py", line 126, in <module>
    saver_to_restore.restore(sess, args.restore_path)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1574, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,512,18] rhs shape= [1,1,512,255]
	 [[{{node save/Assign_291}} = Assign[T=DT_FLOAT, _class=["loc:@yolov3/yolov3_head/Conv_14/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/Conv_14/weights, save/RestoreV2/_583)]]

Caused by op 'save/Assign_291', defined at:
  File "/hdd/YOLOv3_TensorFlow/train.py", line 81, in <module>
    saver_to_restore = tf.train.Saver(var_list=tf.contrib.framework.get_variables_to_restore(include=args.restore_include, exclude=args.restore_exclude))
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1094, in __init__
    self.build()
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1106, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1143, in _build
    build_save=build_save, build_restore=build_restore)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 787, in _build_internal
    restore_sequentially, reshape)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 119, in restore
    self.op.get_shape().is_fully_defined())
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
    validate_shape=validate_shape)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
    use_locking=use_locking, name=name)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3272, in create_op
    op_def=op_def)
  File "/hdd/YOLOv3_TensorFlow/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,512,18] rhs shape= [1,1,512,255]
	 [[{{node save/Assign_291}} = Assign[T=DT_FLOAT, _class=["loc:@yolov3/yolov3_head/Conv_14/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](yolov3/yolov3_head/Conv_14/weights, save/RestoreV2/_583)]]


Process finished with exit code 1

[mew124]

Did you solve the issue? I encountered the same problem.

[alex-razor]

Did you solve the issue? I encountered the same problem.

Not yet..

@wizyoung

[XinghuaJi]

Same issue