Does the OpenVINO execution provider for ONNX Runtime work in ML.NET?

[natelowry]

trafficstars

Is your feature request related to a problem? Please describe.

I'm using an ONNX model via ApplyOnnxModel and some pre-processing (which is really fast!), but I'd like it to use the OpenVINO execution provider instead of just the CPU.

We are maxing out the CPU and not hitting a high enough framerate with just the CPU. CUDA with gpuDeviceId is not an option due to hardware limitations.

I can't figure out a way to make the ONNX Runtime called by ML.NET actually use the OpenVINO execution provider. Is there something simple I'm missing?

Describe the solution you'd like

I'd like the ONNX Runtime used by ML.NET to allow OpenVINO as an execution provider. Maybe that's in the ApplyOnnxModel options as something like OnnxRuntime = OnnxRuntimes.OpenVINO, OnnxRuntime = OnnxRuntimes.CUDA, or OnnxRuntime = OnnxRuntimes.oneDNN, etc.

Describe alternatives you've considered

I can run OpenVINO using the ONNX Runtime Nuget packages built with OpenVINO using AppendExecutionProvider_OpenVINO, but the pre-processing is much slower and the level of abstraction is way lower than I'd like (making the pipeline harder to manage). The inference runs at least 2x faster on the GPU though, so I want to use the GPU via that runtime.

I've tried dropping in all the onnxruntime/openvino dlls to the executable directory to see if it uses those at runtime, but no luck. Do I need to do a custom build of ML.NET?

Additional context

I can provide some more source code or examples if that's helpful, but I think it's more of a concept question than code question.

This is a great project and I appreciate everyone that's working on it.

[natelowry]

Update on this, I'm going to attempt to fork and add the OpenVINO execution provider at session option setup. If I get that working is there any interest in making this a real feature? I'd be happy to submit a PR (that may need work).

I'm thinking the options above might be something like: OpenVINODeviceID = ["CPU_FP32"|"GPU_FP32"|"GPU_FP16"|"MYRIAD_FP16"|"VAD-M_FP16"|"VAD-F_FP32"] or possibly an Enum w/ those values.

Would that be useful? Think this will work? :)

Update: I'd need to use the Intel forked Nuget packages here that are not the latest...so I may abandon the PR idea, but may maintain a fork (if it works)

Here are the steps I did to make it work:

1. clone this repo
2. run build.cmd
3. download the Intel Nuget packages here
4. replace the existing Microsoft.ML.OnnxRuntime.Managed package reference in the Microsoft.ML.OnnxTransformer project with the Intel one
5. add the Intel Microsoft.ML.OnnxRuntime.OpenVino package reference to the Microsoft.ML.OnnxTransformer project
6. add sessionOptions.AppendExecutionProvider_OpenVINO("GPU_FP16"); (or whatever flavor you prefer) to the OnnxModel constructor in OnnxUtils.cs:207 here right before creating the new InferenceSession
7. for some reason the required DLLs don't get pulled in via Nuget, so I have to manually copy over the dlls (and maybe libs) from the OpenVino Nuget package runtimes\win-x64\native folder into the output folder for your project (or the artifacts folder for whatever you're running in this solution) These DLLs and xml are required: onnxruntime.dll, onnxruntime_providers_openvino.dll, openvino.dll, tbb.dll, openvino_onnx_frontend.dll, plugins.xml, openvino_intel_gpu_plugin.dll, openvino_intel_gna_plugin.dll, gna.dll, openvino_intel_cpu_plugin.dll
8. profit 🎉

Hope that's helpful if anyone else wants to do this. Seems like getting this in the project properly may be too much for me to work on right now unfortunately.

NOTE: while you're in there, you might as well disable all ONNX Runtime graph optimizations too per OpenVINO's recommendation. 🚤 GraphOptimizationLevel = GraphOptimizationLevel.ORT_DISABLE_ALL

[luisquintanilla]

Thanks for this investigation and writeup @natelowry. As you pointed out, today there's no easy way to use execution providers like OpenVINO from ML.NET. However, based on your findings, I can see opportunities to simplify the process.

Tagging a few folks to get their thoughts on ways we can simplify the process you went through: @ericstj @eerhardt @michaelgsharp @tarekgh

[michaelgsharp]

If we are just wanting to add the concept of "execution provider" with onnx, we would probably just want to have a way to specify this stuff in our onnx api itself. In this specific case its harder because the whole underlying nuget has to change for OpenVINO, but for any execution providers that ONNX provides out of the box I think we could just modify the api to allow us to specify it.

@luisquintanilla I'm going to add this to future, because we will want to add this at least in some form to take advantage of the performance gains onnx provides with these providers.