ONNX Conversion fails due to unsupported ops: LinSpace and MatrixInverse

[chiragsamal]

I tried converting a TensorFlow model in .pb format to ONNX using tf2onnx. However, the conversion fails for the below 2 ops.

LinSpace and MatrixInverse

ERROR - Tensorflow op [get_homographies_av/MatrixInverse: MatrixInverse] is not supported 2023-02-22 11:59:11,549 - ERROR - Tensorflow op [cost_volume_homography/while/warping_by_homography/LinSpace_1: LinSpace] is not supported 2023-02-22 11:59:11,549 - ERROR - Tensorflow op [cost_volume_homography/while/warping_by_homography/LinSpace: LinSpace] is not supported 2023-02-22 11:59:11,583 - ERROR - Unsupported ops: Counter({'LinSpace': 2, 'MatrixInverse': 1})

I was able to convert MatrixInverse using the --extra_opset com.microsoft:1

!python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset com.microsoft:1

But not sure how to convert LinSpace op.

In this tf2onnx Custom Ops Tutorial by onnxruntime-extensions: https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/tf2onnx_custom_ops_tutorial.ipynb It's written:

Adding a custom op conversion rule using the command line

We need to tell the converter how to convert the TF DecodeGif op. Even if our custom op will have the same name as the TF op, the node must be tagged with the custom ops domain ai.onnx.contrib.

Pass --extra_opset ai.onnx.contrib:1 and --custom-ops DecodeGif:ai.onnx.contrib flags to the converter.

Like this in command line: !python -m tf2onnx.convert --saved-model "saved_model2" --output "model2a.onnx" --extra_opset ai.onnx.contrib:1 --custom-ops DecodeGif:ai.onnx.contrib

I similarly tried to use the same for LinSpace as it is same as Tensorflow op (tf.linspace) like this: --extra_opset ai.onnx.contrib:1 and --custom-ops Linspace:ai.onnx.contrib

But I'm not able to use two different --extra_opset and --custom_ops at a time like this: !python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset ai.onnx.contrib:1 –custom-ops Linspace:ai.onnx.contrib --extra_opset com.microsoft:1 --custom-ops MatrixInverse:com.microsoft

Is there any way I can solve this issue by using two different custom_ops?

[chiragsamal]

Any update @fatcat-z? Eagerly waiting for your reply.

[fatcat-z]

Ideally, you can use a statment like below to include multiple values for --extra_opset and --custom-ops:

!python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset ai.onnx.contrib:1,com.microsoft:1 –custom-ops Linspace:ai.onnx.contrib,MatrixInverse:com.microsoft

Unfortunately, --extra_opset doesn't support it yet. The PR is going to fix it. Before we publish a new release, you might build from source to install tf2onnx and export your model.

[chiragsamal]

I tried this after building it from this pull request #2136 After running this command: !python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset ai.onnx.contrib:1,com.microsoft:1 –-custom-ops Linspace:ai.onnx.contrib,MatrixInverse:com.microsoft

The model got converted suceesfully but I got this error when I loaded the model:

Load model from mvsnet_chirag.onnx failed:Fatal error: com.microsoft:MatrixInverse(-1) is not a registered function/op

Earlier also, I could convert MatrixInverse Custom Op but could not convert LinSpace op.

From this notebook: tf2onnx_custom_ops_tutorial. I am using TF Linspace op and as mentioned in the notebook if our custom op will have the same name as the TF op, the node must be tagged with the custom ops domain ai.onnx.contrib.

Pass --extra_opset ai.onnx.contrib:1 and --custom-ops LinSpace:ai.onnx.contrib flags to the converter.

Please help @fatcat-z

[fatcat-z]

1. The conversion of MatrixInverse has been implemented in tf2onnx, so we can convert it successfully by adding these arguments.
2. The conversion of LinSpace is not done yet in tf2onnx, so you have to follow the steps of "Case 2: Defining new custom ops with Python" in tf2onnx_custom_ops_tutorial.
3. "Fatal error: com.microsoft:MatrixInverse(-1) is not a registered function/op" - This might be a new problem, could you please file another issue so we can have a separated thread to track it?

[chiragsamal]

I believe when we run this command: !python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset ai.onnx.contrib:1,com.microsoft:1 –-custom-ops Linspace:ai.onnx.contrib,MatrixInverse:com.microsoft

During the onnx conversion it is not able to import the MatrixInverse. I believe this is because the after the changes in the PR #2136 the code was only changed to take multiple arguments for the extra opset but it was not able to import the extra opsets which are being passed. Because if you see the MatrixInverse is working perfectly fine if we just import MatrixInverse op from com.microsoft:1 domain but is not being registered when we are taking multiple extra_opset.

[fatcat-z]

You can see the implementation of MatrixInverse here. But no implementation for Linspace yet.

[chiragsamal]

Oh, Ok. In that case then I need to write a custom Op for LinSpace with the help of this tutorial: Example of converting TensorFlow model with custom op to ONNX. After adding the custom operator implementation and registering it in ONNX Runtime then I should run this command: !python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset ai.onnx.contrib:1,com.microsoft:1 –-custom-ops Linspace:ai.onnx.contrib,MatrixInverse:com.microsoft

To make it working, Right?

[fatcat-z]

Oh, Ok. In that case then I need to write a custom Op for LinSpace with the help of this tutorial: Example of converting TensorFlow model with custom op to ONNX. After adding the custom operator implementation and registering it in ONNX Runtime then I should run this command: !python -m tf2onnx.convert --input mvsnet.pb --inputs [in_images:0,in_scaled_cams:0] --outputs out_dataterm_vol:0 --output mvsnet_chirag.onnx --opset 17 --extra_opset ai.onnx.contrib:1,com.microsoft:1 –-custom-ops Linspace:ai.onnx.contrib,MatrixInverse:com.microsoft

To make it working, Right?

Yes.

[chiragsamal]

Hi @fatcat-z As mentioned by you I followed this tutorial to create a custom operator: Linspace.

1. Added the Tensorflow custom operator implementation in C++ and registered it with TensorFlow and get a TensorFlow custom op library lib_linspace.so I have checked this lib_linspace.so the file is working perfectly fine like tf.linspace. This is the gist link of linspace.cc file:
2. Convert the Operator to ONNX I tried to convert the Linspace operator to ONNX using the code with similar functionality: Range Operator present in the controlflow.py with a few changes required for linspace operator. Here you can see the implementation of the op handler according to the op definitions registered it with the @tf_op decorator. This is the gist link of the py file: linspace.py

The main issue is when I run this linspace.py file, it gives the error:

Spec ::::: [TensorSpec(shape=(1,), dtype=tf.float32, name='start'), TensorSpec(shape=(1,), dtype=tf.float32, name='stop'), TensorSpec(shape=(1,), dtype=tf.int32, name='num')] Print output ::::::::::::: Tensor("output:0", shape=(32711,), dtype=float32) Traceback (most recent call last): File "linspace.py", line 147, in onnx.checker.check_model(onnx_model) File "/home/chirag/.local/lib/python3.8/site-packages/onnx/checker.py", line 119, in check_model C.check_model(protobuf_string, full_check) onnx.onnx_cpp2py_export.checker.ValidationError: No Op registered for Linspace with domain_version of 15

==> Context: Bad node spec for node. Name: Linspace OpType: `Linspace

I'm not sure what the issue is. Please help.

[chiragsamal]

Any update @fatcat-z ? Eagerly waiting for your reply. :)

[fatcat-z]

Please refer to here and check the annotation(domain=constants.CONTRIB_OPS_DOMAIN) of ConvertUnsortedSegmentJoinOp function. You have to add a custom domain name for this op, otherwise it will use the default domain which doesn't contain your custom op obviously.

[frytoli]

Hi @chiragsamal -- thanks for sharing your code! Did you ultimately figure out how to register your custom linspace op? I'm hoping to do the same thing and exporting with custom ops is a new concept to me. Thanks in advance!