opencv How to call a saved_model formatted model via opencv

We know that saved_model is often used for tfserving for model deployment. It has the following structure： assets/ assets.extra/ variables/ variables.data-?????-of-????? variables.index saved_model.pb|saved_model.pbtxt

Now I want to call the saved_model format model through opencv，How can I do this, thank you very much.

I know that the .pb model exported by tf1.x can be imported through the structure file .pbtxt，but what should be done with saved_model

The platform is c ++,ubuntu 18.04，opencv4.1

Mar 11 '20 12:03 JCCVW

Please provide a reproducer. Upload the model somewhere so we can try to run it.

Mar 15 '20 06:03 dkurt

Please take a look at https://github.com/opencv/opencv/issues/16879#issuecomment-603416876 if it can help.

Mar 25 '20 10:03 dkurt

Please take a look at #16879 (comment) if it can help.

Thank you very much for your reply. I'll try it and I'll prepare a model for you to test

Mar 25 '20 14:03 JCCVW

Can we close an issue?

Apr 06 '20 10:04 dkurt

Can we close an issue?

I'm sorry for the late reply. It's not easy for me to log in to GitHub. Please give me some more time. I'm preparing the model

Apr 08 '20 05:04 JCCVW

@wangwenchao-job, kind reminder

May 05 '20 08:05 dkurt

@wangwenchao-job, kind reminder

Thank you very much for your reply！

The model is a little big. I didn't find an effective way to upload it, so I sent it to you by email. Did you receive it？

May 05 '20 15:05 JCCVW

@wangwenchao-job, please ping me once again from corresponding email thread. It's hard to find. Or you may try to upload to Dropbox, in example.

May 05 '20 15:05 dkurt

Please take a look at #16879 (comment) if it can help.

How to get pbtxt file after getting pb file, why don't you need pbtxt file here

May 11 '20 11:05 JCCVW

@wangwenchao-job, this is optional file which is not required by default.

May 11 '20 13:05 dkurt

@wangwenchao-job, this is optional file which is not required by default.

I have sent you the model at the address: "Dmitry Kurtaev"[email protected]

I wonder if you can access the download link below： https://www.jianguoyun.com/p/DRuhIW8QtKK2CBjB95cD

I use this method #16879 (comment) to try to load my model，But got the following error： The main errors are: TypeError: Expected argument names ['inputs'] but got values for ['flatten_input']. Missing: ['inputs'].

---------------------------------------------------------------------------------------------------------------The complete information is as follows：

2020-05-12 21:01:25.088723: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll 2.1.0 2020-05-12 21:02:45.902607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll 2020-05-12 21:02:46.916358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1 coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s 2020-05-12 21:02:46.916541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll 2020-05-12 21:02:47.173023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll 2020-05-12 21:02:47.380716: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll 2020-05-12 21:02:47.434142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll 2020-05-12 21:02:47.624859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll 2020-05-12 21:02:47.730697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll 2020-05-12 21:02:48.165401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2020-05-12 21:02:48.454887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0 2020-05-12 21:02:48.491061: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 2020-05-12 21:02:48.495194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1 coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s 2020-05-12 21:02:48.495462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll 2020-05-12 21:02:48.495549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll 2020-05-12 21:02:48.495693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll 2020-05-12 21:02:48.495841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll 2020-05-12 21:02:48.495981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll 2020-05-12 21:02:48.496070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll 2020-05-12 21:02:48.496157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2020-05-12 21:02:48.498013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0 2020-05-12 21:03:04.086203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-05-12 21:03:04.086325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 2020-05-12 21:03:04.086467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N 2020-05-12 21:03:04.105270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2990 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/weights:0' shape=(3, 3, 3, 32) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/gamma:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_mean:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_variance:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/weights:0' shape=(3, 3, 3, 32) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/gamma:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_mean:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_variance:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/weights:0' shape=(3, 3, 3, 32) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/gamma:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_mean:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_variance:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/weights:0' shape=(3, 3, 3, 32) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/gamma:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_mean:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'FeatureExtractor/MobilenetV2/Conv/BatchNorm/moving_variance:0' shape=(32,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables(). WARNING:tensorflow:AutoGraph could not transform <bound method ConcreteFunction.call of <tensorflow.python.eager.wrap_function.WrappedFunction object at 0x0000017653DC0288>> and will run it as-is. Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: No module named 'tensorflow_core.keras' Traceback (most recent call last): File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 909, in get_concrete_function self._initialize(args, kwargs, add_initializers_to=initializers) File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 497, in _initialize *args, **kwds)) File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\function.py", line 2389, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\function.py", line 2703, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\function.py", line 2593, in _create_graph_function capture_by_value=self._capture_by_value), File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\framework\func_graph.py", line 978, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 439, in wrapped_fn return weak_wrapped_fn().wrapped(*args, **kwds) File "D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\framework\func_graph.py", line 968, in wrapper raise e.ag_error_metadata.to_exception(e) TypeError: in converted code:

D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\function.py:1551 __call__
    return self._call_impl(args, kwargs)
D:\anaconda\envs\new_tf2\lib\site-packages\tensorflow_core\python\eager\function.py:1583 _call_impl
    list(set(self._arg_keywords) - set(specified_keywords))))

TypeError: Expected argument names ['inputs'] but got values for ['flatten_input']. Missing: ['inputs'].

May 12 '20 13:05 JCCVW

Please take a look at #16879 (comment) if it can help.

@dkurt

The code i used：

import tensorflow as tf print(tf.version)

from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('saved_model') infer = loaded.signatures['detection_signature']

f = tf.function(infer).get_concrete_function(input='detection_signature') f2 = convert_variables_to_constants_v2(f) graph_def = f2.graph.as_graph_def()

with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f: f.write(graph_def.SerializeToString())

But I don't know how to modify this function： f = tf.function(infer).get_concrete_function(input='detection_signature')

May 12 '20 14:05 JCCVW

Hi! I were able to freeze the graph but it seems unsupported by current version of OpenCV:

import tensorflow as tf
print(tf.__version__)

from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('saved_model')
infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(inputs=tf.TensorSpec(shape=[None, 28, 28, 3], dtype=tf.float32))
f2 = convert_variables_to_constants_v2(f)
graph_def = f2.graph.as_graph_def()

for node in graph_def.node:
   print(node.op, node.name)

# Export frozen graph
with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f:
   f.write(graph_def.SerializeToString())

import numpy as np
import cv2 as cv

net = cv.dnn.readNet('frozen_graph.pb')
inp = np.random.standard_normal([1, 3, 28, 28]).astype(np.float32)
net.setInput(inp)
out = net.forward()
print(out.shape)

error:

error: (-2:Unspecified error) Can't create layer "StatefulPartitionedCall" of type "StatefulPartitionedCall" in function 'getLayerInstance'

May 13 '20 10:05 dkurt

Hi! I were able to freeze the graph but it seems unsupported by current version of OpenCV:

import tensorflow as tf
print(tf.__version__)

from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('saved_model')
infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(inputs=tf.TensorSpec(shape=[None, 28, 28, 3], dtype=tf.float32))
f2 = convert_variables_to_constants_v2(f)
graph_def = f2.graph.as_graph_def()

for node in graph_def.node:
   print(node.op, node.name)

# Export frozen graph
with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f:
   f.write(graph_def.SerializeToString())

import numpy as np
import cv2 as cv

net = cv.dnn.readNet('frozen_graph.pb')
inp = np.random.standard_normal([1, 3, 28, 28]).astype(np.float32)
net.setInput(inp)
out = net.forward()
print(out.shape)

error:

error: (-2:Unspecified error) Can't create layer "StatefulPartitionedCall" of type "StatefulPartitionedCall" in function 'getLayerInstance'

Thank you very much for your reply！

I can successfully export the pb file using the above code, but the file size is only 1k, so is there a problem with this export method?

May 18 '20 09:05 JCCVW

Hi! I were able to freeze the graph but it seems unsupported by current version of OpenCV:

import tensorflow as tf
print(tf.__version__)

from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('saved_model')
infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(inputs=tf.TensorSpec(shape=[None, 28, 28, 3], dtype=tf.float32))
f2 = convert_variables_to_constants_v2(f)
graph_def = f2.graph.as_graph_def()

for node in graph_def.node:
   print(node.op, node.name)

# Export frozen graph
with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f:
   f.write(graph_def.SerializeToString())

import numpy as np
import cv2 as cv

net = cv.dnn.readNet('frozen_graph.pb')
inp = np.random.standard_normal([1, 3, 28, 28]).astype(np.float32)
net.setInput(inp)
out = net.forward()
print(out.shape)

error:

error: (-2:Unspecified error) Can't create layer "StatefulPartitionedCall" of type "StatefulPartitionedCall" in function 'getLayerInstance'

Do you have any suggestions for me? Thank you very much！@dkurt

May 20 '20 07:05 JCCVW

@wangwenchao-job, can you refer source of the model? Is that possible to save it with TensorFlow 1.15.x frozed graph format or TensorFlow 2.x Keras format?

May 20 '20 10:05 dkurt

kind reminder @wangwenchao-job If your problem is solved, please let us know :)

Sep 28 '20 08:09 pwnorbitals

Hi! Have someone solved this problem? I have the similar one. I will be grateful if you share your solution here!

Oct 05 '20 22:10 andidenko

I am in this same spot. Created the frozen_graph.pb from the saved_model.pb with the script you provided(some alterations), and now I have this error after trying to do a forward pass of the model. Can't create layer "StatefulPartitionedCall" of type "StatefulPartitionedCall" in function 'getLayerInstance'

Jan 26 '22 20:01 iaverypadberg

I faced the same error as well. Has anyone found a solution somehow?

Jul 25 '22 13:07 azad96