server
server copied to clipboard
triton-inference-server
Is there a way to load LD_PRELOAD plugins dynamically?
Description I want to load LD_PRELOAD after some computations in docker image but currently I am unable to find a solution about this. I also tried to add lines in nvidia_entrypoint.sh but that solution is also not working.
Triton Information FROM nvcr.io/nvidia/tritonserver:21.03-py3
Are you using the Triton container or did you build it yourself? Using a Container
To Reproduce Steps to reproduce the behavior. In Dockerfile
- FROM nvcr.io/nvidia/tritonserver:21.03-py3
- ENV LD_PRELOAD=$(some bash command)
Expected behavior I want it to load the list of plugins that I have downloaded in /plugins directory but currently I am unable to create there list and attached to environment dynamically. They may have any names so I cannot provide the constant value. It would be a great help if you can tell my where to put these paths in code (e.g. tritonserver lifecycle) so I can resolve this.
@Tabrizian I've tried to edit the nvidia_entrypoint.sh but it's also not setting the environment variables?? Any suggestions?
@sbmalik Did you try with a helper script that returns a string X with all .so's in /plugins with there full absolute paths?
then launch server with LD_PRELOAD=X tritonserver ... ?
@tanmayv25 I am using the docker way and trying to load the plugin in your described way always give me this error that /opt/tritonserver/nvidia_entrypoint.sh: 71: LD_PRELOAD=xyz.so no such file or directory e.t.c.. any other thing before tritonserver command does not work
@sbmalik is the shared library you are trying to load inside the docker? If not, you need to run the docker in interactive mode, copy the library into it and then run the LD_PRELOAD=X tritonserver ... command inside it instead of via the nvidia_entrypoint.sh
@CoderHam that was a nice approach but I can't do this because I am working on a complete product and running the container with other containers via docker-compose. So, manually doing this would not be a solution I think.
@sbmalik copy your plugins into docker images， and setting env LD_PRELOAD=/path/to/plugins in the docker-compose.yaml
@austingg yes this is a good approach, but I am downloading my plugins from AWS, during the building of the docker container. The RUN command is not able to set a variable permanently and we can't use the ENV with command to set the plugins afterwards
Is mounting a directory that contains the plugin while launching your container not an option?
@CoderHam No, because as you know the Triton Inference Server provides a functionality of setting the model repository an AWS S3 bucket so I am loading, unloading all models from there. For these models, plugins are stored on the S3 bucket too. So, while building the container I'm syncing the directory for plugins from S3
Is mounting a directory that contains the plugin while launching your container not an option?
I deploy Triton on Kubernetes, so i want to upload and syncing plugins from s3 too. But it seem impossible to achieve that
Is mounting a directory that contains the plugin while launching your container not an option?
Does Triton support load LD_PRELOAD plugins from S3 dynamically ?
Does Triton support load LD_PRELOAD plugins from S3 dynamically ?
Unfortunately no. The issue with preloading is that it should be performed before triton is launched.
Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue
The TensorFlow backend now allows you to load custom plugins at runtime, in case that helps with the above. We've created similar functionality in some of the other backends as well such as TensorRT.
