RAM memory growth of triton server, until killed by OS

[InfiniteLife]

Im using nvcr.io/nvidia/tritonserver:23.10-py3 container for my inferencing, using C++ GRPC API. There is several models in container, Yolov8-like architecture in Tensorrt plus a few Torchscript models. When inferencing I notice linear growth of RAM memory consumption of triton server, starting from 12-15 GB and linearly growing to up to 80 GB after 12 hours of constant inferencing, and growing further until, seems like, getting killed by OS OOM killer(Ubuntu 22.04).

Triton Server model load mode is default(not explicit), in inferencing API no shared memory is used, Sync and Async calls are made.

My question is what is good way to debug it?

[pvijayakrish]

@InfiniteLife Is it possible to separately run TensorRT and PyTorch models. If the issue goes away for either one, that will help narrow down the issue to a single backend.

[InfiniteLife]

My last experiment with config for all Torchscript models with parameter:

parameters: {
key: "ENABLE_CACHE_CLEANING"
    value: {
    string_value:"true"
    }
}

shown no memory growth. Seems like clearing cache helps. Pytorch is wierd.

[pvijayakrish]

@InfiniteLife Could you please share the steps to reproduce the issue?

[InfiniteLife]

I will need share models for that, but I cannot. But as I mentioned issue is gone if cache clearing is enabled for pytorch.