[QST] Edge compute using Transform4rec models with ONNX runtime

[fire]

❓ Questions & Help

Details

I was wondering if there was a tutorial or a workflow for using Transform4rec models on edge compute using ONNX runtime?

Would be interested to collaborate.

[rnyak]

@fire hello. Thanks for your question. Currently, we do not have an example with ONNX runtime. You can contribute of course if you want to.

[fire]

Any suggestions on how to approach this?

1. Generate a model using transform4rec
2. ???
3. Convert model from torchscript to an onnx model
4. Execute onnx model using onnxruntime on directml (nvidia, intel, amd), cuda directly and cpu.

[rnyak]

@fire that'd be a question for Nvidia Triton team. You can ask your question on Triton repo. Thanks.

[fire]

I cross posted https://github.com/triton-inference-server/server/issues/3976. Feel free to keep this open or close if wanted.

[jobergum]

You can just export the trained torch model to onnx, and use onnx runtime for inference with the same inputs as used during training. See for example this repo for how to do transformer inference in the browser using onnx runtime https://github.com/jobergum/browser-ml-inference