Enable ONNX export for `VisionDecoderEncoderModel`

[lewtun]

🚀 Feature request

As discussed on the forums, it would be nice if one could export VisionDecoderEncoderModel classes using the transformers.onnx package.

Motivation

It is currently not possible to export VisionDecoderEncoderModel classes (or vision models more generally), unless the end user is willing to write their own OnnxConfig. It might make sense to first explore what is involved in exporting ViT before tackling this more complex example.

Looking at this a bit more closely, I can see that transformers.onnx currently has a tight integration with tokenizers (e.g. to generate dummy inputs), so some refactoring will be necessary to support other modalities.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[albertoandreottiATgmail]

I believe progress on this ticket will be welcome by the communitiy.

[lewtun]

Hey @albertoandreottiATgmail thanks for the interest! This is next up on my list now that we've just merged the ViT export in #15658 :)

[IlyasMoutawwakil]

Hey @albertoandreottiATgmail thanks for the interest! This is next up on my list now that we've just merged the ViT export in #15658 :)

Hi @lewtun is there any way (or script) to export trocr to onnx ?

[tanmayjain98]

Hi all, any updates on this? I've also tried to convert TrOCR to ONNX without success. Thanks in advance :) cc @lewtun

[NielsRogge]

Re-opening this issue as it hasn't been resolved yet

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[NielsRogge]

Adding another feature request for this: https://github.com/NielsRogge/Transformers-Tutorials/issues/183

[BakingBrains]

Adding another feature request for this: NielsRogge/Transformers-Tutorials#183

Any updates on this one?

Thank you

[lewtun]

Hi folks, @mht-sharma will be tackling this issue - stay tuned :)

[BakingBrains]

@mht-sharma Any updates?

Thank you.

[lewtun]

@BakingBrains you can see the PR from @mht-sharma linked above - we should be able to get this merged pretty soon!

[BakingBrains]

@lewtun and @mht-sharma Thank you.