server
server copied to clipboard
triton-inference-server
Clients supporting sending multiple synchronous inferences at the same time
Hi, in the doc for java client, it's mentioned that sending multiple synchronous inferences is not supported. I want to make sure whether other clients support it or not. Thanks.
Hi, the limitation should be java client specific (unless explicitly mentioned by other clients). For example, this python grpc client example sends two synchronous inference at line 126 and line 159.
CC @jbkyang-nvi
it's mentioned that sending multiple synchronous inferences is not supported.
@heibaidaolx123 do you mind to point me to the place that this is mentioned? I am not sure if this refers to there can only be one inference at a time even if multiple client objects are used, which can be limitation specific to Java client if that is the case.
@heibaidaolx123 do you mind to point me to the place that this is mentioned? I am not sure if this refers to there can only be one inference at a time even if multiple client objects are used, which can be limitation specific to Java client if that is the case.
Hi, @GuanLuo, it's mentioned here.
@heibaidaolx123 it just means it doesn't support InferMulti like the C++ client does. We can update the documentation to make it more clear.
What is your use case? Do you want to send parallel infer requests using different clients? Do you want to send a batch of infer requests like in InferMulti? Are you using Java client? If not, I would recommend refer to the C++/Python client documentation instead. The Java client is being slowly worked on and has not reached parity with python/C++ clients
@jbkyang-nvi Thanks for your advice. I tried to use multiple Java client in multiple threads to send parallel infer requests, and I got GC errors. Then I turned to use C++ client, and it works as expected.
I tried to use multiple Java client in multiple threads to send parallel infer requests, and I got GC errors.
@dyastremsky @jbkyang-nvi is this expected when using Java client?
@heibaidaolx123 can you send a code snippet & gc error? It should not happen. We will let you know when we fix it
Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this.
