Deployment of Seamless M4T Model - Exporting text.decoder to ONNX or Using torch.jit.trace

[HesamAlavian]

Description

I am currently working on deploying the Seamless M4T model for text-to-text translation on a Triton server. I have successfully exported the text.encoder to ONNX and traced it using torch.jit.trace. However, I am encountering issues when trying to export the text.decoder to ONNX or trace it using torch.jit.trace. The process is being blocked by IncrementalStateBag.

Steps to Reproduce

1. Exporting text.encoder:

   • Successfully exported to ONNX.
   • Successfully traced using torch.jit.trace.

2. Exporting text.decoder:

   • Attempted to export to ONNX.
   • Attempted to trace using torch.jit.trace.
   • Encountered an issue with IncrementalStateBag.

Observed Behavior

• When attempting to export or trace the text.decoder, the process is interrupted by IncrementalStateBag, preventing successful completion.

Expected Behavior

• The text.decoder should be successfully exported to ONNX or traced using torch.jit.trace, similar to the text.encoder, enabling deployment on the Triton server for text-to-text translation.

Environment

• Model: Seamless M4T
• Export Methods Attempted: ONNX export, torch.jit.trace
• Server: Triton Inference Server

Questions and Assistance Needed

1. Is there a recommended approach to handle IncrementalStateBag when exporting or tracing the text.decoder?
2. Are there any workarounds or specific configurations that I should consider to successfully export the text.decoder?
3. Has anyone successfully deployed the Seamless M4T model on Triton server, and if so, could you provide guidance or a reference implementation?

Any assistance or guidance on this matter would be greatly appreciated. Thank you!

---

Additional Information

Please let me know if there is any additional information required to diagnose the issue.

Thank you for your support!