ValueError: Tensor adanet/iteration_0/ensemble_t0_3_self_built_cnn/weighted_subnetwork_0/subnetwork/batch_normalization_2/moving_mean is not found in /data4/data/adanet/modelZoo/2018_12_10_11_23/model.ckpt-7500 checkpoint

[Npccc]

Hello, I am using tf 1.12.0,Python 2.7.15，I run my own built subnetwork under the fashion mnist dataset，but I am getting the warning:

Traceback (most recent call last): File "class_builtCNN.py", line 39, in steps=None)) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate return executor.run() File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 610, in run return self.run_local() File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 711, in run_local saving_listeners=saving_listeners) File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 499, in train self._prepare_next_iteration(input_fn) File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 618, in _prepare_next_iteration params) File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 578, in _call_adanet_model_fn self._adanet_model_fn(features, labels, mode, params) File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 1093, in _adanet_model_fn tf.train.warm_start(latest_checkpoint, vars_to_warm_start=[".*"]) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/warm_starting_util.py", line 463, in warm_start _warm_start_var(variable, ckpt_to_initialize_from, prev_var_name) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/warm_starting_util.py", line 181, in _warm_start_var checkpoint_utils.init_from_checkpoint(prev_ckpt, {prev_tensor_name: var}) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/checkpoint_utils.py", line 187, in init_from_checkpoint _init_from_checkpoint, ckpt_dir_or_file, assignment_map) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/distribute.py", line 1053, in merge_call return self._merge_call(merge_fn, *args, **kwargs) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/distribute.py", line 1061, in _merge_call return merge_fn(self._distribution_strategy, *args, **kwargs) File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/checkpoint_utils.py", line 216, in _init_from_checkpoint tensor_name_in_ckpt, ckpt_dir_or_file, variable_map ValueError: Tensor adanet/iteration_0/ensemble_t0_3_self_built_cnn/weighted_subnetwork_0/subnetwork/batch_normalization_2/moving_mean is not found in /data4/data/adanet/modelZoo/2018_12_10_11_23/model.ckpt-7500 checkpoint

Any suggestions on how to solve the problem or avoid it in the first place? Thank you so much @cweill

[cweill]

Interesting. Are you using tf.keras.layers or tf.layers? I think I ran into this issue with tf.keras.layers because they handle variable scoping differently.

[Npccc]

You are right，thank you @cweill

[cweill]

Thank you, for confirming @Npccc. We'll look into this.