version-checker icon indicating copy to clipboard operation
version-checker copied to clipboard

version-checker seemingly leaks memory and gets oom-killed

Open nadiamoe opened this issue 4 years ago • 6 comments

I am running version-checker on a single node, quite small cluster with ~60 pods. So far it is working nicely, but I do not understand the memory behavior it has.

I'm basically running the sample deployment file, plus the --test-all-containers flag and some cpu limits:

        resources:
          requests:
            cpu: 10m
            memory: 32M
          limits:
            cpu: 50m
            memory: 128M

kubectl get pod -o yaml

Over time, I see that version-checker approaches the memory limit and then stays near ~99% for a while. After some time, the kernel kills the ct due to OOM and k8s restarts the pod.

Memory chart

However, I do not see anything alarming in the logs, other than some failures and expected permission errors.

This doesn't seem to have any functional impact, but does fire some alerts and doesn't look good on my dashboards :)

Is this behavior intended, and/or is there any way to prevent it?

nadiamoe avatar Feb 20 '21 11:02 nadiamoe

We are seeing a similar behaviour while running Version Checker. Would be interested to know if there are recommended values for the limits?

trastle avatar Feb 24 '21 02:02 trastle

Also seeing something similar with Version Checker getting OOM killed fairly frequently.

Trede1983 avatar Mar 15 '21 21:03 Trede1983