ipex-llm
ipex-llm copied to clipboard
[Nano] improve OpenVINO user experience
Description
bigdl.nano.openvino.OpenVINOModel
has now provided limited features for users and few notifications about current status. We can only simply run inference like:
from bigdl.nano.openvino import OpenVINOModel
openvino_model = OpenVINOModel("resnet18-xnor-binary-onnx-0001.xml")
preds = openvnino_model.forward_step(x)
This is the only thing we can do right now. Here we can discuss about more features we can provide to improve user experience
Device setting
Similar to Pytorch, we can provide a function for users to simply copy models from one device to another.
openvino_cpu_model = OpenVINOModel("resnet18-xnor-binary-onnx-0001.xml", device='CPU')
openvino_gpu_model = openvino_cpu_model.to('GPU')
preds = openvnino_model.forward_step(x)
Notification/Log
Notify users of model information.
openvino_model = OpenVINOModel("resnet18-xnor-binary-onnx-0001.xml")
will output:
[INFO] File Path : resnet18-xnor-binary-onnx-0001.xml
[INFO] Precision: FP32
[INFO] Graph has 1 input and 2 outputs.
[INFO] Device: CPU
Extensions
Add extensions to OpenVINO Core. Seems if we want to support more devices, we need to load extra extensions. For example, NCS needs extra extension.
Needs further investigation.