gramineproject
Failure to run encrypted models on openvino/model_server:latest due to Memory Fault
Description of the problem
I was running openvino/model_server with encryption using gsc. I encrypted the model files using gramine-sgx-pf-crypt and added the corresponding entries to the manifest.template file, such as { path = "/mnt/tmpfs/model_encrypted", uri = "file:/mnt/tmpfs/model_encrypted", type = "encrypted" }. Additionally, I included fs.insecure__keys.default = "88f1c4333d2d1e85a81d3b72f92fb396" in the manifest.template file.
I was successfully able to run openvino/model_server:2023.0, and the OpenVINO server started without issues. However, when I built the latest openvino image openvino/model_server:latest and tried the same setup, it failed with a memory fault debug: memory fault at 0x0000cc95 (IP = 0x39b6f85e2).
I have attached the manifest file for openvino/model_server, and you can download the model by following these steps:
mkdir models curl --create-dirs https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.xml https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.bin -o models/1/face-detection-retail-0004.xml -o models/1/face-detection-retail-0004.bin
Steps to reproduce
I have attached the manifest file for openvino/model_server, and you can download the model by following these steps:
mkdir models curl --create-dirs https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.xml https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.bin -o models/1/face-detection-retail-0004.xml -o models/1/face-detection-retail-0004.bin
Expected results
openvino/model_server:latest should be able to run successfully with encrypted model
Actual results
openvino/model_server:latest is failing with memory fault
I have attached the log files as well
This memory fault issue is reproducible on bare metal by downloading the ovms 2024.5 binary from docs.openvino.ai for ubuntu 24.04 Attaching the files for both manifest.template files, gramine-direct and gramine-sgx logs logs.zip
This memory fault issue is reproducible on bare metal
IIUC, the issue also applies to core Gramine. If so, could you pls raise a discussion/issue in the core Gramine repo?
This memory fault issue is reproducible on bare metal
IIUC, the issue also applies to core Gramine. If so, could you pls raise a discussion/issue in the core Gramine repo?
Raised on issue on gramine - https://github.com/gramineproject/gramine/issues/2086