Performance issues with onnx_tf

[Terizian]

I have developed a model on Matlab and saved it using the onnx framework. The size of the model is 25 MB. I am currently trying to move this model to production using the supported libraries on Python. In my code, I have the following:

model = onnx.load(onnx_path)
tf_rep = prepare(model)  

This takes 25.71 seconds to run, making it very heavy for production usage.

Additionally, when running the predictions:

output = tf_rep.run(x)

Each prediction takes on average 4 seconds to run. My target is running 100 predictions in a second and I'm finding that impossible with the framework. What are things that I may try to speed it up?

[prasanthpul]

Are you constrained to use TF to run your model?

[Terizian]

Not at all! However, I tried some other options before and this was the only one that managed to run for me. Do you recommend other frameworks?

[prasanthpul]

Do you know what opset version your model is exported in? We've seen good performance on our models with https://github.com/Microsoft/onnxruntime. It supports opset 7 and higher. If your model is less than that, you may be able to upgrade it using https://github.com/onnx/onnx/blob/master/docs/OpsetVersionConverter.md

[Terizian]

I'm not exactly certain but I'll try it out and let you know if it improves on the time :) thanks a lot!

[prasanthpul]

Let me know how it goes. If it continues to show poor performance, we likely need to take a look at whether the model was exported in an efficient way.

[Terizian]

The model I'm running has opset 6 so I tried upgrading it using the Python tutorial. Here's the code I tried:

import onnx
from onnx import version_converter, helper

onnx_path = 'model.onnx'
model = onnx.load(onnx_path)
converted_model = version_converter.convert_version(model, 7)

It throws this error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-13-4607f0053661> in <module>()
      6 # https://github.com/onnx/onnx/blob/master/onnx/version_converter.py#L21
      7 # Apply the version conversion on the original model
----> 4 converted_model = version_converter.convert_version(model, 7)
      9 
     10 print('The model after conversion:\n{}'.format(converted_model))

C:\Anaconda3\lib\site-packages\onnx\version_converter.py in convert_version(model, target_version)
    164         raise ValueError('VersionConverter only accepts int as target_version, incorrect type: {}'.format(type(target_version)))
    165     model_str = model.SerializeToString()
--> 166     converted_model_str = C.convert_version(model_str, target_version)
    167     return onnx.load_from_string(converted_model_str)

RuntimeError: Inferred shape and existing shape differ in rank: (2) vs (4)

[prasanthpul]

@Terizian the code shows version 7 but the output shows version 8. is the output from the right run?

@houseroad I believe you worked on the version converter. can you elaborate on the issue being run into here? "Inferred shape and existing shape differ in rank: (2) vs (4)" would ideally mention the node having the issue and a bit more detail about the failure

[Terizian]

Apologies for the mistake, I was trying several versions. I have updated my earlier post with the error.

[prasanthpul]

@Terizian BTW, if you still want to pursue the onnx_tf route in parallel in the meantime, I suggest you file an issue in the onnx-tensorflow repo so the authors of that converter can take a look.

[Terizian]

Personally, I am not too picky with the framework used as long as it's light and fast for a production environment. I'll take your suggestion and post there. Thanks a lot :-)

[Terizian]

@Terizian the code shows version 7 but the output shows version 8. is the output from the right run?

@houseroad I believe you worked on the version converter. can you elaborate on the issue being run into here? "Inferred shape and existing shape differ in rank: (2) vs (4)" would ideally mention the node having the issue and a bit more detail about the failure

Any idea why the conversion fails? I would like to work on speeding up the code using onnxruntime.

[saptarshim-dal]

Hi,

Referenced this issue from https://github.com/onnx/tutorials/issues/90.

I am facing an exactly identical issue while upgrading onnx model from opset 6 to opset 7. Is it at all possible?

Below is the stack trace: converted_model = version_converter.convert_version(original_model, 7)

(op_type:Softmax, name:prob): Inferred shape and existing shape differ in rank: (2) vs (4)

RuntimeError Traceback (most recent call last) in ----> 1 converted_model = version_converter.convert_version(original_model, 7)

~/anaconda3/envs/Inference_Server/lib/python3.7/site-packages/onnx/version_converter.py in convert_version(model, target_version) 164 raise ValueError('VersionConverter only accepts int as target_version, incorrect type: {}'.format(type(target_version))) 165 model_str = model.SerializeToString() --> 166 converted_model_str = C.convert_version(model_str, target_version) 167 return onnx.load_from_string(converted_model_str)

RuntimeError: Inferred shape and existing shape differ in rank: (2) vs (4)

Thanks.