onnxruntime icon indicating copy to clipboard operation
onnxruntime copied to clipboard

Create script to export BART encoder and decoder for use with custom beam search op

Open BowenBao opened this issue 3 years ago • 5 comments

Is your feature request related to a problem? Please describe. Under https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/models, a set of tools to enable users to export BART encoder and decoder to ONNX, for use with custom beam search op.

BowenBao avatar Apr 13 '22 20:04 BowenBao

ORT folks contributed an example for this on the transformers repository if it can be useful: https://github.com/huggingface/transformers/tree/main/examples/research_projects/onnx/summarization

mfuntowicz avatar Apr 14 '22 10:04 mfuntowicz

@BowenBao, thanks for raising the feature request. we are working on encoder and decoder support in beam search op (currently using T5 as example). After that is ready, we will work on BART integration.

tianleiwu avatar Apr 15 '22 18:04 tianleiwu

@tianleiwu thanks for following up. I have created a simple initial version for helping Ye with the implementation. Will create a PR afterwards.

BowenBao avatar Apr 18 '22 16:04 BowenBao

PR work in progress. https://github.com/microsoft/onnxruntime/pull/11629

BowenBao avatar May 25 '22 17:05 BowenBao

8/11 Update:

  1. Model has been exported successfully.
  2. Model has mismatched performance between Pytorch and ONNX
  3. Validated Encoder and Decoder parts of model, and the results match Pytorch
  4. Now, Having ONNXRUNTIME team checks beam search op result

titaiwangms avatar Aug 11 '22 17:08 titaiwangms