aimet
aimet copied to clipboard
Published
20 hours ago •
quic
quic
Error while using AIMET TensorFlow Cross Layer Equalization APIs
Hello everyone,
I started exploring AIMET library and I installed AIMET using docker, when i started trying to implement CLE from the documentation https://quic.github.io/aimet-pages/AimetDocs/api_docs/tensorflow_cross_layer_equalization.html#tf-cle . I am using the resnet50 model for 10 class classification. I am getting errors while calling the function cross_layer_equalization_auto() . My code is given below:
def cross_layer_equalization_auto():
""" perform auto cross layer equalization """
# load a model
tf.keras.backend.clear_session()
model_resnet = ResNet50(weights='imagenet', input_shape=(224, 224, 3),include_top=False)
x=model_resnet.output
x=tf.keras.layers.GlobalAveragePooling2D()(x)
pred=tf.keras.layers.Dense(10,activation='softmax')(x)
model_f=tf.keras.Model(inputs=model_resnet.input, outputs=pred)
# model_f.summary()
sess = tf.compat.v1.Session()
op = sess.graph.get_operations()
# get starting op name to invoke api for cle
input_op_name = 'input_1'
output_op_name = 'dense/Softmax'
# Equalize a model with Batchnorms
# Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
# use the new session returned for further evaluations on TF graph
with sess.as_default():
sess.run(tf.global_variables_initializer())
new_session = equalize_model(sess, input_op_name, output_op_name)
sess.close()
My error when i call the fucntion is shown below:
2023-11-02 07:09:41,995 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,996 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,996 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,997 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,997 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,998 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,998 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,999 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,999 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,000 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,000 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,001 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,001 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,001 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,002 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,002 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,003 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,003 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,004 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,004 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,006 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,006 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,006 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,007 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,007 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,007 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,008 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,008 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,009 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,009 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,009 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,010 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,010 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,010 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,011 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,011 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,012 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,012 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,012 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block4_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,013 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,013 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block4_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,013 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,014 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block4_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,014 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,014 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,015 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,015 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,015 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,016 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,016 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,017 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,017 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,018 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,018 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,018 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,019 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,019 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,020 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,022 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,023 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,023 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,024 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,024 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,024 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block4_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block4_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,026 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block4_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,026 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,027 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block5_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,027 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,028 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block5_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,028 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,029 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block5_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,029 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,029 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block6_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,030 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,030 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block6_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,031 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,031 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block6_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,031 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,032 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,033 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,033 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,034 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,035 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,035 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,037 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,037 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,037 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,039 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,039 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,039 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42.732657: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.732804: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.732866: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.732964: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.733024: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.733077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 648 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-02_07:09:42.252434/temp
2023-11-02 07:09:43,125 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-02_07:09:42.252434/temp
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1378, in BaseSession._do_call(self, fn, *args)
1377 try:
-> 1378 return fn(*args)
1379 except errors.OpError as e:
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1361, in BaseSession._do_run.<locals>._run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
1360 self._extend_graph()
-> 1361 return self._call_tf_sessionrun(options, feed_dict, fetch_list,
1362 target_list, run_metadata)
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1454, in BaseSession._call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
1452 def _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list,
1453 run_metadata):
-> 1454 return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
1455 fetch_list, target_list,
1456 run_metadata)
InvalidArgumentError: 2 root error(s) found.
(0) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
[[{{node Placeholder_3}}]]
[[ReadVariableOp_3/_641]]
(1) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
[[{{node Placeholder_3}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
InvalidArgumentError Traceback (most recent call last)
Cell In[8], line 1
----> 1 cross_layer_equalization_auto()
Cell In[7], line 31, in cross_layer_equalization_auto()
29 with sess.as_default():
30 sess.run(tf.global_variables_initializer())
---> 31 new_session = equalize_model(sess, input_op_name, output_op_name)
32 sess.close()
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:842, in equalize_model(sess, start_op_names, output_op_names)
839 start_op_names = [start_op_names]
841 # fold batchnorm layers
--> 842 after_bn_fold_sess, folded_pairs = fold_all_batch_norms(sess, start_op_names, output_op_names)
844 # replace any ReLU6 layers with ReLU
845 graph_util = GraphSearchUtils(after_bn_fold_sess.graph, start_op_names, output_op_names)
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:364, in fold_all_batch_norms(sess, input_op_names, output_op_names)
361 pairs_to_return.append((pair[0], pair[1].op))
363 # Convert the standalone BNs which are not folded
--> 364 bn_converted = convert_standalone_batchnorms(after_fold_sess, input_op_names, output_op_names, bns_to_fold)
365 if bn_converted:
366 logger.info("%d BatchNorms' weights got converted", len(bn_converted))
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:394, in convert_standalone_batchnorms(sess, input_op_names, output_op_names, bns_folded)
392 for op in list_of_ordered_ops:
393 if op.type in ['FusedBatchNormV3', 'FusedBatchNorm', 'BatchNormalization'] and op not in bns_folded:
--> 394 convert_batchnorm_parameters(sess, op)
395 converted_bns.append(op)
396 logger.debug("%s weights got converted", op)
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:407, in convert_batchnorm_parameters(sess, op)
400 def convert_batchnorm_parameters(sess, op):
401 """
402 Convert the weights of BN such that it works as y = weights * x + bias
403
404 :param sess: TF Session in which the graph is loaded
405 :param op: bn_op which whose weights need to be converted
406 """
--> 407 bn_params = _get_bn_params(sess, op)
408 weight = np.array(bn_params.gamma) / np.array(bn_params.runningVar)
409 bias = np.array(bn_params.beta) - np.array(bn_params.runningMean) * weight
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:211, in _get_bn_params(sess, bn)
208 with sess.graph.as_default():
209 # create BNParams type and populate
210 bn_params = libpymo.BNParams()
--> 211 bn_params.beta = BNUtils.get_beta_as_numpy_data(sess, bn).reshape(-1)
212 bn_params.gamma = BNUtils.get_gamma_as_numpy_data(sess, bn).reshape(-1)
213 bn_params.runningMean = BNUtils.get_moving_mean_as_numpy_data(sess, bn).reshape(-1)
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/utils/op/fusedbatchnorm.py:267, in BNUtils.get_beta_as_numpy_data(sess, bn_op)
264 beta_tensor = BNUtils.get_beta_read_var_op_tensor(sess.graph, bn_op)
266 with sess.graph.as_default():
--> 267 numpy_data = sess.run(beta_tensor)
269 return numpy_data
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:968, in BaseSession.run(self, fetches, feed_dict, options, run_metadata)
965 run_metadata_ptr = tf_session.TF_NewBuffer() if run_metadata else None
967 try:
--> 968 result = self._run(None, fetches, feed_dict, options_ptr,
969 run_metadata_ptr)
970 if run_metadata:
971 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1191, in BaseSession._run(self, handle, fetches, feed_dict, options, run_metadata)
1188 # We only want to really perform the run if fetches or targets are provided,
1189 # or if the call is a partial run that specifies feeds.
1190 if final_fetches or final_targets or (handle and feed_dict_tensor):
-> 1191 results = self._do_run(handle, final_targets, final_fetches,
1192 feed_dict_tensor, options, run_metadata)
1193 else:
1194 results = []
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1371, in BaseSession._do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1368 return self._call_tf_sessionprun(handle, feed_dict, fetch_list)
1370 if handle is None:
-> 1371 return self._do_call(_run_fn, feeds, fetches, targets, options,
1372 run_metadata)
1373 else:
1374 return self._do_call(_prun_fn, handle, feeds, fetches)
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1397, in BaseSession._do_call(self, fn, *args)
1392 if 'only supports NHWC tensor format' in message:
1393 message += ('\nA possible workaround: Try disabling Grappler optimizer'
1394 '\nby modifying the config for creating the session eg.'
1395 '\nsession_config.graph_options.rewrite_options.'
1396 'disable_meta_optimizer = True')
-> 1397 raise type(e)(node_def, op, message)
InvalidArgumentError: Graph execution error:
Detected at node 'Placeholder_3' defined at (most recent call last):
Node: 'Placeholder_3'
Detected at node 'Placeholder_3' defined at (most recent call last):
Node: 'Placeholder_3'
2 root error(s) found.
(0) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
[[{{node Placeholder_3}}]]
[[ReadVariableOp_3/_641]]
(1) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
[[{{node Placeholder_3}}]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'Placeholder_3':
I am unable to interpret this error can anyone help me in understanding what is the problem and how to correct the issue, that would be a great help for me. Thank you in advance.
Hi @sandeep1404 In TF 2.x, eager execution is enabled by default and you are using AIMET TensorFlow CLE API designed for Graph mode (uses tf.Session/tf.Graph). You will need to disable eager execution in order to use it. To disable eager execution, please put tf.compat.v1.disable_eager_excution() API before instantiating the model.
import tensorflow as tf
tf.compat.v1.disable_eager_execution()
Also, if you are interested in using TF eager execution mode, you can also take a look at AIMET Keras CLE API as well which is designed for eager execution.
https://quic.github.io/aimet-pages/releases/latest/api_docs/keras_cross_layer_equalization.html#api-keras-cle
@quic-hitameht , Thanks for the reply but I already disabled eager execution and run the code, after disabling eager execution only i am getting the above error. Can you please check that once. Thank you in advance.
@sandeep1404 Could you please share your complete script to reproduce the issue? Thank you.
Hi @quic-hitameht , Thanks for the response, PFB my code the code is running fine but i cannot able to understand whats going inside the CLE, I tried to interpret the logs but i couldnt get information from it, I tried visualizing the new session graph in tensorboard, but the graph_cle folder is being created for the new session and when i open tensorboard with the logdir as graph_cle using the command tensorboard --logdir=graphs_cle it gives the following logs and when i open the localhost link http://localhost:6006/ it says This site can’t be reached
2023-11-03 05:47:16.784452: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-03 05:47:16.851571: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-03 05:47:16.868884: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-03 05:47:17.148442: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-11-03 05:47:17.148500: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-11-03 05:47:17.148503: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-11-03 05:47:17.526458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 05:47:17.531403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 05:47:17.531544: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
W1103 05:47:17.557937 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.559506 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.565727 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.568787 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.570783 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.572910 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.574856 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.575885 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.577856 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.578965 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.580727 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.582290 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.10.1 at http://localhost:6006/ (Press CTRL+C to quit)
can you please kindly check the code and report whats the issue and how to understand whats going inside. Thankyou in advance.
Here is my code
# tensorflow
import tensorflow as tf
tf.compat.v1.disable_eager_execution()
from tensorflow.keras.applications.resnet50 import ResNet50
# Cross layer Equalization related imports
from aimet_tensorflow.cross_layer_equalization import equalize_model
# def cross_layer_equalization_auto():
""" perform auto cross layer equalization """
# load a model
tf.compat.v1.keras.backend.clear_session()
model_resnet = ResNet50(weights='imagenet', input_shape=(224, 224, 3))
sess = tf.compat.v1.keras.backend.get_session()
# op=sess.graph.get_operations()
# print(op[-1])
# get starting op name to invoke api for cle
input_op_name = 'input_1'
output_op_name = 'init'
# Equalize a model with Batchnorms
# Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
# use the new session returned for further evaluations on TF graph
with sess.as_default():
new_session = equalize_model(sess, input_op_name, output_op_name)
tf_tensorboard_writer = tf.compat.v1.summary.FileWriter('graphs_cle', new_session.graph)
# op_new=new_session.graph.get_operations()
# print(op_new[2])
# [print(m.values()) for m in op_new]
tf_tensorboard_writer.close()
sess.close()
Here are the logs of the outputs:
2023-11-03 04:56:41.819730: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-03 04:56:41.821519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:41.821685: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:41.821770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.178780: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179428: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
2023-11-03 04:56:42.268941: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
WARNING:tensorflow:From /tmp/ipykernel_3416/1158554977.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 04:56:43,737 - tensorflow - WARNING - From /tmp/ipykernel_3416/1158554977.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 04:56:44.702115: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702340: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702475: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-03_04:56:43.974220/temp
2023-11-03 04:56:45,042 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-03_04:56:43.974220/temp
2023-11-03 04:56:46.537949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538108: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538171: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538292: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538357: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./replace_relu6_with_relu_2023-11-03_04:56:45.693136/temp
2023-11-03 04:56:46,867 - tensorflow - INFO - Restoring parameters from ./replace_relu6_with_relu_2023-11-03_04:56:45.693136/temp
2023-11-03 04:56:48.083233: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083843: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083919: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_cls_2023-11-03_04:56:47.416744/temp
2023-11-03 04:56:48,633 - tensorflow - INFO - Restoring parameters from ./temp_cls_2023-11-03_04:56:47.416744/temp
2023-11-03 04:56:48,984 - CrossLayerEqualization - ERROR - High Bias folding is not supported for models without BatchNorm Layers
@sandeep1404 Thanks for sharing your script. equalize_model API expects list of input and output names of given model. In your script, you have provided incorrect output_op_name. For tf.keras.applications.ResNet50(), it should be predictions/Softmax.
Regarding your question to understand CLE, the feature attempts to equalize weight ranges across different channels. It is beneficial for models where weight ranges for different channels in a layer has wide variance. You can use the returned session for further evaluations on TF graph.
Hi @quic-hitameht , I changed the output_op_name to predictions/Softmax but when i run the code it has some key error, here are the logs when i change the output name:
WARNING:tensorflow:From /tmp/ipykernel_8450/2048079406.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 09:00:06,897 - tensorflow - WARNING - From /tmp/ipykernel_8450/2048079406.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 09:00:07,116 - Utils - INFO - ...... subset to store [conv1_conv/Conv2D, conv1_bn]
2023-11-03 09:00:07,116 - Utils - INFO - ...... subset to store [conv1_conv/Conv2D, conv1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_1_conv/Conv2D, conv2_block1_1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_1_conv/Conv2D, conv2_block1_1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_2_conv/Conv2D, conv2_block1_2_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_2_conv/Conv2D, conv2_block1_2_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv2_block1_3_conv/Conv2D, conv2_block1_3_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv2_block1_3_conv/Conv2D, conv2_block1_3_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv3_block1_1_conv/Conv2D, conv3_block1_1_bn]
2023-11-03 09:00:07,119 - Utils - INFO - ...... subset to store [conv3_block1_1_conv/Conv2D, conv3_block1_1_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_2_conv/Conv2D, conv3_block1_2_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_2_conv/Conv2D, conv3_block1_2_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_3_conv/Conv2D, conv3_block1_3_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv3_block1_3_conv/Conv2D, conv3_block1_3_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_1_conv/Conv2D, conv4_block1_1_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_1_conv/Conv2D, conv4_block1_1_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_2_conv/Conv2D, conv4_block1_2_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_2_conv/Conv2D, conv4_block1_2_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_3_conv/Conv2D, conv4_block1_3_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_3_conv/Conv2D, conv4_block1_3_bn]
2023-11-03 09:00:07,123 - Utils - INFO - ...... subset to store [conv5_block1_1_conv/Conv2D, conv5_block1_1_bn]
2023-11-03 09:00:07,123 - Utils - INFO - ...... subset to store [conv5_block1_1_conv/Conv2D, conv5_block1_1_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_2_conv/Conv2D, conv5_block1_2_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_2_conv/Conv2D, conv5_block1_2_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_3_conv/Conv2D, conv5_block1_3_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block1_3_conv/Conv2D, conv5_block1_3_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block3_1_conv/Conv2D, conv5_block3_1_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block3_1_conv/Conv2D, conv5_block3_1_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_2_conv/Conv2D, conv5_block3_2_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_2_conv/Conv2D, conv5_block3_2_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_3_conv/Conv2D, conv5_block3_3_bn]
2023-11-03 09:00:07,127 - Utils - INFO - ...... subset to store [conv5_block2_1_conv/Conv2D, conv5_block2_1_bn]
2023-11-03 09:00:07,127 - Utils - INFO - ...... subset to store [conv5_block2_1_conv/Conv2D, conv5_block2_1_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_2_conv/Conv2D, conv5_block2_2_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_2_conv/Conv2D, conv5_block2_2_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_3_conv/Conv2D, conv5_block2_3_bn]
2023-11-03 09:00:07,129 - Utils - INFO - ...... subset to store [conv5_block1_0_conv/Conv2D, conv5_block1_0_bn]
2023-11-03 09:00:07,129 - Utils - INFO - ...... subset to store [conv4_block6_1_conv/Conv2D, conv4_block6_1_bn]
2023-11-03 09:00:07,130 - Utils - INFO - ...... subset to store [conv4_block6_1_conv/Conv2D, conv4_block6_1_bn]
2023-11-03 09:00:07,133 - Utils - INFO - ...... subset to store [conv4_block6_2_conv/Conv2D, conv4_block6_2_bn]
2023-11-03 09:00:07,133 - Utils - INFO - ...... subset to store [conv4_block6_2_conv/Conv2D, conv4_block6_2_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block6_3_conv/Conv2D, conv4_block6_3_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block5_1_conv/Conv2D, conv4_block5_1_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block5_1_conv/Conv2D, conv4_block5_1_bn]
2023-11-03 09:00:07,135 - Utils - INFO - ...... subset to store [conv4_block5_2_conv/Conv2D, conv4_block5_2_bn]
2023-11-03 09:00:07,135 - Utils - INFO - ...... subset to store [conv4_block5_2_conv/Conv2D, conv4_block5_2_bn]
2023-11-03 09:00:07,136 - Utils - INFO - ...... subset to store [conv4_block5_3_conv/Conv2D, conv4_block5_3_bn]
2023-11-03 09:00:07,136 - Utils - INFO - ...... subset to store [conv4_block4_1_conv/Conv2D, conv4_block4_1_bn]
2023-11-03 09:00:07,137 - Utils - INFO - ...... subset to store [conv4_block4_1_conv/Conv2D, conv4_block4_1_bn]
2023-11-03 09:00:07,137 - Utils - INFO - ...... subset to store [conv4_block4_2_conv/Conv2D, conv4_block4_2_bn]
2023-11-03 09:00:07,138 - Utils - INFO - ...... subset to store [conv4_block4_2_conv/Conv2D, conv4_block4_2_bn]
2023-11-03 09:00:07,138 - Utils - INFO - ...... subset to store [conv4_block4_3_conv/Conv2D, conv4_block4_3_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_1_conv/Conv2D, conv4_block3_1_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_1_conv/Conv2D, conv4_block3_1_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_2_conv/Conv2D, conv4_block3_2_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block3_2_conv/Conv2D, conv4_block3_2_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block3_3_conv/Conv2D, conv4_block3_3_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block2_1_conv/Conv2D, conv4_block2_1_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_1_conv/Conv2D, conv4_block2_1_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_2_conv/Conv2D, conv4_block2_2_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_2_conv/Conv2D, conv4_block2_2_bn]
2023-11-03 09:00:07,142 - Utils - INFO - ...... subset to store [conv4_block2_3_conv/Conv2D, conv4_block2_3_bn]
2023-11-03 09:00:07,143 - Utils - INFO - ...... subset to store [conv4_block1_0_conv/Conv2D, conv4_block1_0_bn]
2023-11-03 09:00:07,143 - Utils - INFO - ...... subset to store [conv3_block4_1_conv/Conv2D, conv3_block4_1_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_1_conv/Conv2D, conv3_block4_1_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_2_conv/Conv2D, conv3_block4_2_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_2_conv/Conv2D, conv3_block4_2_bn]
2023-11-03 09:00:07,145 - Utils - INFO - ...... subset to store [conv3_block4_3_conv/Conv2D, conv3_block4_3_bn]
2023-11-03 09:00:07,146 - Utils - INFO - ...... subset to store [conv3_block3_1_conv/Conv2D, conv3_block3_1_bn]
2023-11-03 09:00:07,146 - Utils - INFO - ...... subset to store [conv3_block3_1_conv/Conv2D, conv3_block3_1_bn]
2023-11-03 09:00:07,147 - Utils - INFO - ...... subset to store [conv3_block3_2_conv/Conv2D, conv3_block3_2_bn]
2023-11-03 09:00:07,147 - Utils - INFO - ...... subset to store [conv3_block3_2_conv/Conv2D, conv3_block3_2_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block3_3_conv/Conv2D, conv3_block3_3_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block2_1_conv/Conv2D, conv3_block2_1_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block2_1_conv/Conv2D, conv3_block2_1_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_2_conv/Conv2D, conv3_block2_2_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_2_conv/Conv2D, conv3_block2_2_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_3_conv/Conv2D, conv3_block2_3_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block1_0_conv/Conv2D, conv3_block1_0_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_1_conv/Conv2D, conv2_block3_1_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_1_conv/Conv2D, conv2_block3_1_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_2_conv/Conv2D, conv2_block3_2_bn]
2023-11-03 09:00:07,151 - Utils - INFO - ...... subset to store [conv2_block3_2_conv/Conv2D, conv2_block3_2_bn]
2023-11-03 09:00:07,151 - Utils - INFO - ...... subset to store [conv2_block3_3_conv/Conv2D, conv2_block3_3_bn]
2023-11-03 09:00:07,152 - Utils - INFO - ...... subset to store [conv2_block2_1_conv/Conv2D, conv2_block2_1_bn]
2023-11-03 09:00:07,152 - Utils - INFO - ...... subset to store [conv2_block2_1_conv/Conv2D, conv2_block2_1_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_2_conv/Conv2D, conv2_block2_2_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_2_conv/Conv2D, conv2_block2_2_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_3_conv/Conv2D, conv2_block2_3_bn]
2023-11-03 09:00:07,154 - Utils - INFO - ...... subset to store [conv2_block1_0_conv/Conv2D, conv2_block1_0_bn]
2023-11-03 09:00:07.588003: W tensorflow/c/c_api.cc:349] Operation '{name:'conv1_relu/Relu' id:77 op device:{requested: '', assigned: ''} def:{{{node conv1_relu/Relu}} = Relu[T=DT_FLOAT](conv1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:08.147625: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_1_relu/Relu' id:221 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:08.936135: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_2_relu/Relu' id:292 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:09.711713: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_add/add' id:363 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_0_bn/cond/Identity, conv2_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:10.443266: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_add/add' id:363 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_0_conv/BiasAdd:0, conv2_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:11.163123: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_1_relu/Relu' id:435 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:11.954976: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_2_relu/Relu' id:506 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:12.688118: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_add/add' id:577 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_out/Relu, conv2_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:13.299646: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_1_relu/Relu' id:649 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:13.896197: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_2_relu/Relu' id:720 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:14.492094: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_add/add' id:791 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_add/add}} = AddV2[T=DT_FLOAT](conv2_block2_out/Relu, conv2_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:15.089717: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_1_relu/Relu' id:933 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:15.713069: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_2_relu/Relu' id:1004 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:16.363501: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_add/add' id:1075 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_0_bn/cond/Identity, conv3_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:17.035267: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_add/add' id:1075 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_0_conv/BiasAdd:0, conv3_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:17.673926: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_1_relu/Relu' id:1147 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:18.419134: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_2_relu/Relu' id:1218 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:19.247085: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_add/add' id:1289 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_out/Relu, conv3_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:20.006269: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_1_relu/Relu' id:1361 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:20.701548: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_2_relu/Relu' id:1432 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:21.338198: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_add/add' id:1503 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_add/add}} = AddV2[T=DT_FLOAT](conv3_block2_out/Relu, conv3_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:22.048276: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_1_relu/Relu' id:1575 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block4_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:22.755388: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_2_relu/Relu' id:1646 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block4_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:23.630489: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_add/add' id:1717 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_add/add}} = AddV2[T=DT_FLOAT](conv3_block3_out/Relu, conv3_block4_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:24.498999: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_1_relu/Relu' id:1869 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:25.486446: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_2_relu/Relu' id:1940 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:26.267102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_add/add' id:2021 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_0_bn/cond/Identity, conv4_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:27.074924: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_add/add' id:2021 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_0_conv/BiasAdd:0, conv4_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:27.814852: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_1_relu/Relu' id:2093 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:28.713874: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_2_relu/Relu' id:2164 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:29.490330: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_add/add' id:2245 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_out/Relu, conv4_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:30.251893: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_1_relu/Relu' id:2317 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:31.101687: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_2_relu/Relu' id:2388 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:31.850806: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_add/add' id:2469 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_add/add}} = AddV2[T=DT_FLOAT](conv4_block2_out/Relu, conv4_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:32.613102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_1_relu/Relu' id:2541 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block4_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:33.422959: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_2_relu/Relu' id:2612 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block4_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:34.179514: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_add/add' id:2693 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_add/add}} = AddV2[T=DT_FLOAT](conv4_block3_out/Relu, conv4_block4_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:34.946318: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_1_relu/Relu' id:2765 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block5_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:35.775102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_2_relu/Relu' id:2836 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block5_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:36.532354: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_add/add' id:2917 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_add/add}} = AddV2[T=DT_FLOAT](conv4_block4_out/Relu, conv4_block5_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:37.313127: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_1_relu/Relu' id:2989 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block6_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:38.166129: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_2_relu/Relu' id:3060 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block6_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:38.963474: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_add/add' id:3141 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_add/add}} = AddV2[T=DT_FLOAT](conv4_block5_out/Relu, conv4_block6_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:39.889524: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_1_relu/Relu' id:3293 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:41.284167: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_2_relu/Relu' id:3364 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:42.306595: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_add/add' id:3445 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_0_bn/cond/Identity, conv5_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:43.572133: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_add/add' id:3445 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_0_conv/BiasAdd:0, conv5_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:44.796112: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_1_relu/Relu' id:3517 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:46.258673: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_2_relu/Relu' id:3588 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:47.485528: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_add/add' id:3669 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_out/Relu, conv5_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:48.566980: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_1_relu/Relu' id:3741 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:50.032718: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_2_relu/Relu' id:3812 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:51.098022: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_add/add' id:3893 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_add/add}} = AddV2[T=DT_FLOAT](conv5_block2_out/Relu, conv5_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-03_09:00:51.304753/temp
2023-11-03 09:00:52,618 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-03_09:00:51.304753/temp
INFO:tensorflow:Restoring parameters from ./replace_relu6_with_relu_2023-11-03_09:00:53.247088/temp
2023-11-03 09:00:54,638 - tensorflow - INFO - Restoring parameters from ./replace_relu6_with_relu_2023-11-03_09:00:53.247088/temp
INFO:tensorflow:Restoring parameters from ./temp_cls_2023-11-03_09:01:30.907458/temp
2023-11-03 09:01:33,173 - tensorflow - INFO - Restoring parameters from ./temp_cls_2023-11-03_09:01:30.907458/temp
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[3], line 19
15 # Equalize a model with Batchnorms
16 # Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
17 # use the new session returned for further evaluations on TF graph
18 with sess.as_default():
---> 19 new_session = equalize_model(sess, input_op_name, output_op_name)
20 tf_tensorboard_writer = tf.compat.v1.summary.FileWriter('graphs_cle2', new_session.graph)
21 # op_new=new_session.graph.get_operations()
22 # print(op_new[2])
23 # [print(m.values()) for m in op_new]
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:853, in equalize_model(sess, start_op_names, output_op_names)
849 after_cls_sess, cls_set_info_list = CrossLayerScaling.scale_model(after_relu_replace_sess, start_op_names,
850 output_op_names)
852 # high-bias fold
--> 853 after_hbf_sess = HighBiasFold.bias_fold(after_cls_sess, folded_pairs, cls_set_info_list)
855 return after_hbf_sess
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:745, in HighBiasFold.bias_fold(sess, folded_pairs, cls_set_info_list)
732 """
733 Folds bias values greater than 3 * sigma to next layer's bias
734
(...)
739
740 """
742 with sess.graph.as_default():
743
744 # refresh the references saved during bn fold and cls.
--> 745 cls_set_info_list, bn_layers = HighBiasFold._refresh_layer_set_info_before_hbf(sess, folded_pairs,
746 cls_set_info_list)
748 if not bn_layers:
749 logger.error('High Bias folding is not supported for models without BatchNorm Layers')
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:721, in HighBiasFold._refresh_layer_set_info_before_hbf(sess, folded_pairs, cls_set_info_list)
718 # update info saved during batchnorm fold
719 for conv_bn in folded_pairs:
720 # get the new op ref from it's name
--> 721 bn_dict[conv_bn[0].name] = dict_names_to_tf_ops[conv_bn[1].name]
723 # update info saved during cls
724 ClsSetInfo.map_cls_sets_to_new_session(dict_names_to_tf_ops, cls_set_info_list)
KeyError: 'conv1_bn/cond/Identity'
can you please kindly check the Key error, i tried changing in cross_layer_equalization.py file but still the error persists. Thank you in advance.
@sandeep1404 this definitely looks like a bug. let me take a look. Thanks for reporting this.
Thank you @quic-hitameht for the response, suggest me the solution once if you fixed it. Thank you in advance.
Hi @sandeep1404, I am getting an error at Batchnorm same as yours. Have you solve this bug yet? If yes, please guide me how to solve it.
