server
server copied to clipboard
triton-inference-server
How to maintain confidentiality of models in local deployments
How do I ensure that my ML models are hidden from others in a local deployment of triton inference server. Since we need model files in the model repository, it's easily accessible.
If I use a S3 bucket as model repo, it's still can be downloaded.
How to deploy triton inference server on premise locally, and ensure models can't be accessed or downloaded by the client?
You can use the model repository agents to add custom operations that will be called when the model is loading/unloading.
How to deploy triton inference server on premise locally, and ensure models can't be accessed or downloaded by the client?
Right now, there is no way to download the model files from the client.
Thanks for replying @Tabrizian I would like to explain my issue a bit. My client wants an on-prem local deployment on his computer. I don't want my ML learning models to be accessible to him. In Triton's deployment, the models are generally kept in a directory, which can be easily copied and reused somewhere else without my knowledge. Is there any standardized way to solve this? If I have hosted the models on s3, I'll have to keep s3 credentials on his local system which is again another problem.
@Tabrizian I have a similar problem. In the Repository Agent, It was introduced as 'You can introduce your own code to perform authentication, decryption, conversion, or similar operations when a model is loaded.' But the example 'checksum_repository_agent' is not enough for me.I didn't have enough experience in the server field. Can you provide some simple examples to show 'You can introduce your own code to perform authentication, decryption, conversion, or similar operations when a model is loaded.'
I think model repository agents that Experienced people don't need it, and inexperienced people don't know how to use it
In the checksum_repository_agent GitHub repo, the model can be loaded successful after remove ModelRepositoryAgents section in config. What I expect is : if the MD5 is wrong, the model loading will fail. if there isn't a ModelRepositoryAgents section in the config, the model loading will fail.
@sourabh-burnwal I think you can encrypt the model and store the encrypted models in the model repository and use the model repository agent to decrypt the model when the model is being loaded. However, I think you still need to store the decrypted model on the filesystem so this might be an issue for your scenario (@GuanLuo can correct me here).
@ZJU-lishuang This test might also be helpful to demonstrate how to run the example end-to-end. https://github.com/triton-inference-server/server/blob/main/qa/L0_repoagent_checksum/test.sh Can you explain where you were having trouble using the checksum repository agent so that we can improve the documentation?
@Tabrizian thanks. Yes, I was thinking of doing the same. Can you share any recommended encryption for onnx, python and torchscript models?
@Tabrizian I have run my model for checksum. But when I delete the ModelRepositoryAgents section in the config.txt, the model still load Successfully. What should I do to make the model loading failed when I delete the ModelRepositoryAgents section in the config.txt.
As an option, it is possible to fork your preferred backend and introduce a decryption function directly inside model loading.
@sourabh-burnwal Sorry for the delayed response. We don't have any specific examples for ONNX, Python, and torchscript models but you should be able to use repository agents for the same.
But when I delete the ModelRepositoryAgents section in the config.txt, the model still load Successfully. What should I do to make the model loading failed when I delete the ModelRepositoryAgents section in the config.txt.
I don't think there is a way right now without modifying your backend code. (cc @GuanLuo who has more information).
Hi @ZJU-lishuang, I would like to ask a question please :) you modified your backend by editing files like (e.g. for TensorFlow Backend):
tensorflow/cc/saved_model/reader.cc
tensorflow/cc/saved_model/loader.cc
correct? So basically you need to write some C functions in those files in order to decrypt the model before loading and then (as stated in the docs) rebuild the Backend using /opt/tensorflow/nvbuild.sh etc, am I right?
Thank you in advance!
Closing issue due to lack of activity for Triton development team. Please re-open the issue if you would like to follow up with this issue
@ZJU-lishuang , hi , I have the same issues and was wondering if you could give me pointers as to how and where you adapted the backend? I'm not too familiar with c++ but I looked into the common backend functions and believethe ReadTextFile function in backends_common.cc could be adapted to feature de-cryption, as for the Tensorrt backend the Loadplan Function ,aside from error handling, seems to refer to ReadTextFile.. If it's possible for you to share some insight I'd be very greatful. Thanks!
