nebuly

nebuly copied to clipboard

Hello,

First of all, congratulations, it's a really nice library with a lot of potential and I wish that it will become a standard and help many people deploy models in production.

I was wondering if you plan to support ONNX. I used it in the past as :

• It can help to train models (only with gpu, linux and pytorch for now...)
• It is usable in many other languages as there are bindings for onnx in java / rust / C / python / nodejs etc...

Supporting onnx would help to deploy models on many other languages and platforms. I don't know however how openVino or other formats are, regarding loading optimized models and doing inference on it on other platforms.

Thanks in advance and thanks for your work ! Have a great day.

Hi @Ierezell, and thank you so much for your comment!We currently support the ONNX format, but just in the background! To be more specific, the PyTorch and TensorFlow models are currently converted to ONNX files before being optimized.

We will also provide more in-depth documentation for using the different modules within nebullvm in future releases, but it may also be useful to expose the API directly and add an "optimize_onnx_model" function, similar to what we are doing with other frameworks

Hi @morgoth95, Ok as I understood you're converting all models to onnx and then compile it with any optimizer and check which one is the best.

I don't know how it's possible to do inference with other models (tvm/openvino) as I never played with it but I guess it's then the user to do so as this lib is only doing the compilation step (and that's perfect).

Maybe having the intermediate onnx model outputted as well could be a nice feature for those who want to use the pre-made libraries to do inference on Onnx models (and we can always do custom integration of any .tar / tvm /openvino models).

Thanks for the fast reply, Have a great day