The recent provider/session refactor is not working for OpenVINO.

[jingzhaoou]

I changed GetSessionOptionsImpl() in session.cc to use OpenVINO.

  Ort::SessionOptions sess_opts;
  sess_opts.SetIntraOpNumThreads(num_threads);
  sess_opts.SetInterOpNumThreads(num_threads);
  sess_opts.SetGraphOptimizationLevel(ORT_DISABLE_ALL);
  OrtOpenVINOProviderOptions options;
  options.device_type = "CPU_FP32";
  sess_opts.AppendExecutionProvider_OpenVINO(options);
  return sess_opts;

After that, I ran into errors like

2023-05-16 15:37:00.162490913 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may o
r may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-05-16 15:37:00.162554167 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignmen
ts.
2023-05-16 15:37:00.573335241 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may o
r may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-05-16 15:37:00.573379616 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignmen
ts.
terminate called after throwing an instance of 'Ort::Exception'
  what():  Encountered unknown exception in Run()

I looked at the OnlineZipformerTransducerModel class:

  Ort::SessionOptions sess_opts_;
  Ort::AllocatorWithDefaultOptions allocator_;

While sess_opts_ is initialized using sess_opts_(GetSessionOptions(config)), allocator_ is probably not using the updated session options. That may be the root cause of the above issue.

I appreciate your help on this issue. Thanks a lot!

[jingzhaoou]

Just looked at some OpenVINO sample code. Using Ort::AllocatorWithDefaultOptions should be fine. Need to dig further.

https://github.com/microsoft/onnxruntime-inference-examples/blob/cc2f5efed2e8b4cfe39e23a385f9e4d374ff1910/c_cxx/OpenVINO_EP/Linux/squeezenet_classification/squeezenet_cpp_app.cpp#L225

[csukuangfj]

@jingzhaoou Did you change anything else?

I think you need to download a version of onnxruntime that has openvino enabled.

For instance, if you are using Linux (please change the URLs accordingly if you are using Windows), you can use

wget https://files.pythonhosted.org/packages/04/11/e942b42b074fbbfcb6439a4c1b480ca348057cd2fb19fd4241c0d4bc31cb/onnxruntime_openvino-1.14.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

unzip onnxruntime_openvino-1.14.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

After unzipping, you will find

 tree onnxruntime/capi/
onnxruntime/capi/
├── __init__.py
├── _ld_preload.py
├── _pybind_state.py
├── libonnxruntime_providers_openvino.so
├── libonnxruntime_providers_shared.so
├── libopenvino.so.2230
├── libopenvino_auto_plugin.so
├── libopenvino_c.so.2230
├── libopenvino_hetero_plugin.so
├── libopenvino_intel_cpu_plugin.so
├── libopenvino_intel_gpu_plugin.so
├── libopenvino_onnx_frontend.so.2230
├── libtbb.so.2
├── libtbbmalloc.so.2
├── onnxruntime_collect_build_info.py
├── onnxruntime_inference_collection.py
├── onnxruntime_pybind11_state.cpython-38-x86_64-linux-gnu.so
├── onnxruntime_validation.py
├── plugins.xml
└── training
    └── __init__.py

1 directory, 20 files

By the way, please use https://github.com/k2-fsa/sherpa-onnx/pull/153 as a reference.

You need to link .so you downloaded from PyPI and also please have a look at the comment https://github.com/k2-fsa/sherpa-onnx/pull/153#discussion_r1192401898

[jingzhaoou]

@csukuangfj thanks so much for the info. I will check out the wheel you shared with me.

I was able to compile both OpenVINO and ONNXRuntime supporting OpenVINO from source. The offline flow is working fine with OpenVINO. I encountered errors only in the streaming (online) flow. I suspect somehow in the streaming flow is loading data into CPU.