How to convert SDMG-R Pytorch model to ONNX?

[anuj-rathore]

Deployment for Key Information Extraction

[gaotongxiao]

We don't have a plan for this currently. Ping @AllentDan to see if the deployment team will take this into their future plan. You are also welcome to give it a try and share your progress with our community, if any. :)

[anuj-rathore]

@AllentDan, are you guys looking into it?

[AllentDan]

Hi, @anuj-rathore. SDMG-R is not in our short-term plan. But just like @gaotongxiao said, we welcome that you can give a try on that. Regards.

[amitbcp]

@AllentDan can you provide some pointers on how to start and go about it for SDMGR model ?

Because the framework is building from multiple hooks, I was kinda stuck from where to convert. Some input could help me to create a PR for this

[AllentDan]

Hi, @amitbcp. We have example codes about how to deploy some detection and recognition models in mmocr. Dive into tools/deployment/pytorch2onnx.py and add some codes to make codes execute to torch.onnx.export SDMGR model. Here we probably encounter some unexpected errors. We may use dichotomy to locate the codes that arose errors. To this end, we can use if torch.onnx.is_in_onnx_export() to control whether to run codes that we want to replace with the original codes.

[amitbcp]

Hi @AllentDan !

can you provide some more pointers on how to go forward with the conversion.. even the ONNX inference for SDMGR. Or how you guys do it as a process and I can take it and try to debug it from there?

It will help me move faster and SDMGR will help a lot for community. Looking for your help and guidance

[gouttham]

+1

[AllentDan]

Hi @AllentDan !

can you provide some more pointers on how to go forward with the conversion.. even the ONNX inference for SDMGR. Or how you guys do it as a process and I can take it and try to debug it from there?

It will help me move faster and SDMGR will help a lot for community. Looking for your help and guidance

Let's see, the KIE model needs image tensor, bbox, and text. As onnx does not support string variable input, the text has to be converted to index value and the bbox can be converted to a tensor. You can pass a tuple of these three tensors I guess to torch.onn.export and change the codes inside the KIE model to make it exportable.

That's all I can tell you in detail by now I guess. Best wishes.

[amitbcp]

Thanks @AllentDan , will try it out. Thanks

[gouttham]

Hi @AllentDan I was able to convert the mode into the .onnx file, but I faced an issue during loading this model in onnxruntime

on investigating further, I found that the below line in SDMGRHead.py is causing trouble

nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1)

any suggestions on how to resolve this? can you suggest any alternate snippet ? The onnx model checker has disregarded this, I will raise an issue over there too. Thanks.

[gouttham]

Hi @AllentDan I was able to convert the mode into the .onnx file, but I faced an issue during loading this model in onnxruntime

on investigating further, I found that the below line in SDMGRHead.py is causing trouble

nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1)

any suggestions on how to resolve this? can you suggest any alternate snippet ? The onnx model checker has disregarded this, I will raise an issue over there too. Thanks.

Actually I modified valid = all_nums > 0 as valid = torch.where(all_nums>0)[0] and was able to convert the model into onnx. Thanks

[AllentDan]

Hi, @gouttham. Good job. The line nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1) is too long to analyse codes. Maybe we can split it to several short lines. BTW, you can pull a request to mmocr and tag a WIP so that we can debug together.

[gouttham]

Hi, @gouttham. Good job. The line nodes[valid] = rnn_nodes[valid].gather(1, (all_nums[valid] - 1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, rnn_nodes.size(-1))).squeeze(1) is too long to analyse codes. Maybe we can split it to several shot lines. BTW, you can pull a request to mmocr and tag a WIP so that we can debug together.

Sure I will create a pr. The actual problem is onnx does not support boolean based indexing ie, valid = all_nums > 0. The issue has been resolved by modifying valid = all_nums > 0 to valid = torch.where(all_nums>0)[0], So we need not worry about the above long line :)

Thanks for the reply. Nice repo :)

[syan-dev]

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

[gouttham]

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

[xinsuinizhuan]

how to convert the onnx? I conver the

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

could give the expoterd the onnx download link?

[lyc728]

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

[lyc728]

Hi @ amitbcp, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

[skywalkerfmc]

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

Which pytorch version are you using for convert to onnx? When I set "do_constant_folding=True" in the process of convert to onnx, I got a segmentation fault in pytorch1.7.1.

[gouttham]

Hi @skywalkerfmc , Kindly find below with the version i am using. torch==1.10.0 onnx==1.10.1 onnxruntime==1.6.0

[Bourne-M]

我参考mmocr，写了一个简单版本的，转onnx的时候有些问题，有人能看看吗？谢谢 https://github.com/Bourne-M/SDMG_R

[gaotongxiao]

@Bourne-M Could you elaborate on the issue encountered and the steps to reproduce?

[Bourne-M]

@Bourne-M Could you elaborate on the issue encountered and the steps to reproduce?

好的。 1.sdmg2pb.py->torch2pb 函数中定义了5个输入，但是导出的onnx中只有4个。 2.参照mmocr的实现，代码里有些对tensor的slice操作，在转onnx的时候会引发 warning。

[Bourne-M]

@AllentDan could you help me

[MohamedElghazi01]

Nov 12, 2021

Hi please did you put the script of converting the sdgmr model ?

[MohamedElghazi01]

Hi @gouttham, can you provide me with the code to convert the KIE model to onnx format? Thank you so much!

Sorry for late reply tofuai. Give me few days, i will raise a PR. Thanks.

hi Hi please could you put the script of converting the sdgmr model ?