Trawinski, Dariusz
Trawinski, Dariusz
As of now openvino_backend doesn't support model with a range in the input and output shapes. That is the case for model person-detection-0303. Is has on output the shapes boxes([0~100],5)...
If your model doesn't have named outputs, the following code can add the names: ``` import openvino as ov from pathlib import Path ov_model = ov.Core().read_model(Path(ir_model_dir + "/model.xml")) for idx,...
@nnshah1 there were some changes in the model format to align the API with the models like from pytorch which allows outputs to be without a name. They can be...
Can you include in tests swapping the model version while running the inferences?
@stevegrubb we are setting the version in Makefile https://github.com/openvinotoolkit/model_server/blob/main/Makefile#L178 I'll update it also in the dockerfile to be consistent.
default driver version was set in the dockerfile
@dkalinowski we are compiling tokenizers from source so actually we could use here that version instead of relying on pip packages. Alternatively we could update the default python version to...
@HumerousGorgon could you include the content of graph.pbtxt and complete log?
@HumerousGorgon your config looks correct. Is your model quantized to int4 precision? Can you reduce KV cache size from 6GB to 1GB?
@HumerousGorgon export script was recently corrected to allow continuos batching with HETERO https://github.com/openvinotoolkit/model_server/blob/releases/2025/3/demos/common/export_models/export_model.py Have you exported the model in INT4 precision? I suspect that maybe it is related to the...