W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g0/fw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint

[fdmsouza]

I am trying to run this benchmark using a compiled version of the dev branch. The execution environment consists of python 3.5 and tensorflow 1.2 compiled with MKL support. I have tried to run it with different version of tensorflow 1.1, 1.2, 1.3 and 1.4 (w/ MKL support) -- neither of them work. I now keep getting the following error. Any ideas?

Loading saved model from out/basic/00/save/basic-20000 2017-11-08 23:08:23.116497: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/prepro/u1/bw/bw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/prepro/u1/bw/bw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116558: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/prepro/u1/fw/fw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/prepro/u1/fw/fw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116629: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/prepro/u1_1/bw/bw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/prepro/u1_1/bw/bw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116648: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/prepro/u1_1/fw/fw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/prepro/u1_1/fw/fw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116677: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/main/g0/bw/bw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/main/g0/bw/bw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116696: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/main/g0/fw/fw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/main/g0/fw/fw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116720: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/main/g1/bw/bw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/main/g1/bw/bw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116748: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/main/g1/fw/fw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/main/g1/fw/fw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116768: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/main/g2/bw/bw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/main/g2/bw/bw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.116786: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'model_0/main/g2/fw/fw/while/TensorArrayReadV3/Enter' because the input edge from 'model_0/main/g2/fw/fw/TensorArray_1' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0 2017-11-08 23:08:23.188586: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g0/fw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.187997: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g1/bw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.188589: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key emb/emb_var/char_emb_mat/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.194202: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_1/gate/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.194272: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_0/gate/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.194418: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g0/bw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.195744: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g0/fw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.195849: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key emb/emb_var/word_emb_mat/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.196282: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g0/bw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.196483: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key emb/char/conv/xx/conv1d_5/filter/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.197594: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g1/fw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.198806: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_0/gate/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.199848: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_1/trans/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.200034: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g1/bw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.201510: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_0/trans/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.202920: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g1/fw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.203291: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key emb/char/conv/xx/conv1d_5/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.203856: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g2/bw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.204548: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_1/trans/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.204651: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g2/fw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.206205: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_1/gate/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.206394: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/logits1/first/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.207807: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key highway/highway_network/layer_0/trans/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.208788: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g2/fw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.209054: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/p0/bi_attention/u_logits/first/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.209420: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/logits1/first/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.210972: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key prepro/u1/fw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.211670: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key prepro/u1/fw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.211794: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key prepro/u1/bw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.211952: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/logits2/first/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.212048: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/logits2/first/bias/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.212942: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/p0/bi_attention/u_logits/first/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.213097: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key main/g2/bw/basic_lstm_cell/kernel/ExponentialMovingAverage not found in checkpoint 2017-11-08 23:08:23.213140: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key prepro/u1/bw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint Traceback (most recent call last): File "/home/fillipe/anaconda3/envs/tf3.5mkl/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1139, in _do_call return fn(*args) File "/home/fillipe/anaconda3/envs/tf3.5mkl/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1121, in _run_fn status, run_metadata) File "/home/fillipe/anaconda3/envs/tf3.5mkl/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/home/fillipe/anaconda3/envs/tf3.5mkl/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.NotFoundError: Key main/g0/fw/basic_lstm_cell/bias/ExponentialMovingAverage not found in checkpoint [[Node: save_1/RestoreV2_15 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_15/tensor_names, save_1/RestoreV2_15/shape_and_slices)]]

[ioana-blue]

I'm getting a similar error. I'm using the dev branch and TF 1.1