How to load plugin dynamically

[leanrdchen00918]

My model(not ONNX) uses a plugin that is compiled as a .so file. According to this, it's clear that I should set LD_LIBRARY_PATH when starting up triton server to make my model work.

However, the limitation here is that I have to specify the exact library to link to BEFORE start up. How can I load my model that uses a plugin AFTER I have invoked the server(without shutting it down and restarting it)?

[krishung5]

Hi @leanrdchen00918, unfortunately right now we cannot load those custom plugin/shared libraries at runtime. We can only specify LD_LIBRARY_PATH and LD_PRELOAD variables before starting up Triton. I have filed an enhancement ticket for this. CC @GuanLuo

[GuanLuo]

Yes, LD_PRELOAD is the current way to load custom plugins / ops for most frameworks. To load plugin dynamically, it largely depends on whether the framework libraries expose features (like for ONNX), we will look into it as possible enhancement for the backends.

[dyastremsky]

We have added custom plugins for other backends like TensorFlow and TensorRT. You can see how this was done here and here.

Since some backends now support this, I'm going to close out this ticket.. The implementation would be different for each backend. If you need custom plugins implemented for a specific backend, please let us know.