Working Set of manager container of source-controller pod continuously growth

[syurevich]

We observed that the Working Set (extracted by Prometheus metric container_memory_working_set_bytes) of manager container of source-controller pod continuously growth by about 15MB per 24h.

In several weeks container gets killed as it reaches resource memory limit.

We tried to analyze it with Go’s profiling tool pprof. It does not show any significant memory increase over last 2 weeks.

The "busybox" top command also does not show any significant difference which could explain 15MB/day growth.

On August 26

$ top
Mem total:16198820 anon:1568936 map:1073748 free:3266676
 slab:1160372 buf:668700 cache:9167932 dirty:248 write:0
Swap total:0 free:0

  PID     VSZ  VSZRW   RSS (SHR) DIRTY (SHR) STACK COMMAND
    1   5445m   114m  129m    20 72384     0   132 source-controller ...

On August 27

$ top
Mem total:16198820 anon:1559148 map:1073532 free:3277528
 slab:1166448 buf:668700 cache:9180052 dirty:356 write:56
Swap total:0 free:0

  PID     VSZ  VSZRW   RSS (SHR) DIRTY (SHR) STACK COMMAND
    1   5445m   114m  129m    20 72836     0   132 source-controller ...

Do you have an idea what could cause such a growth of the Working Set?