Ahmet Pokerce

Hi @krishung5 , sorry for the delay, I could not find the shared valgrind output (full output). I am attaching below valgrind result we mainly interested in possibly lost part...

@krishung5 We use A40 GPU, most of the models are tensorrt converted and some small PyTorch models. OOM issue will raise up around 6 days with around 60-80 consecutive requests...

Hi @krishung5 , thanks for detailed answer. However, it seems it is capping at some point but it is not. It slows down in growing but eventually it will grow....

Hi @krishung5 , unfortunately we are not able to change to http directly but I will try the production with perf_analyzer using http client and get back to you. Last...

Hi @krishung5, the 24.02 version of Tritonserver seems promising in terms of RAM grow. I will do more tests and see if the grow problem is resolved there. I will...

There is 0.1 g grow per day with 24.02 with flags. It has been 5 days perf analyzer working there is no cut-off yet with grpc with http I did...

Hi @krishung5 , I was not able to test out 24.03. For 24.02, we will see whether we will get OOM but results will not be available soon. For 24.02...

Hi @krishung5 , I run the grpc and http for several days. I guess your period is not same as my tests so in a day I also do not...

With HTTP client we did not see grow for 2 days. Waiting on production tests to see whether we will see OOM issue.