[help] how much memory is needed for virt-launcher internal usage

[fanlix]

I use harvesterhci for vm hosting. Some heavy load vm get cgroup-oom-killed many many times.

by checking the cgroup file /sys/fs/cgroup/memory/kubepods/burstable/pod12914c4e-282a-4f69-bd3b-f75cba6e715e/memory.limit_in_bytes, i found

• 64G vmi , cgroup mem limit = 69034745856 = 64.2936G (+300M)
• 32G vmi, cgroujp mem limit = 34619084800 = 32.2415G (+247M) (by the way , how is this extra 300M/247M space added)

this 32G vmi pod actual use 34604867584 = 32.228G (from cgroup.xxx.memory.usage_in_bytes) exec into vmi pod, ps show:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 5620588 73640 ?       Ssl  02:41   0:16 /usr/bin/virt-launcher --qemu-timeout 299s --name qc-xx-82 --uid 1bc0bb4a-521a-48eb-af4c-e1d78d81176a
root        25  0.1  0.0 5625324 96384 ?       Sl   02:41   0:34 /usr/bin/virt-launcher --qemu-timeout 299s --name qc-xx-82 --uid 1bc0bb4a-521a-48eb-af4c-e1d78d81176a
root        40  0.0  0.0 1862984 48472 ?       Sl   02:41   0:12 /usr/sbin/libvirtd -f /var/run/libvirt/libvirtd.conf
root        41  0.0  0.0 141132 14856 ?        S    02:41   0:02 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf
qemu        90 27.4  6.3 34539432 33413948 ?   Sl   02:41 124:58 /usr/bin/qemu-system-x86_64 -name guest=qc_qc-xx-82,debug-threads=on -S -object secret,id=masterKey0,
root      1074  0.1  0.0  16832  6000 pts/0    Ss   10:17   0:00 /bin/bash
root      1101 75.0  0.0  38184  4020 pts/0    R+   10:17   0:00 ps aux

• guest-os use 33413948 = 31.866G
• virt-launcher tasks use 73640+96384+48472+14856=233352 = 227.88M
• exec into pod debugging use 16832+38184=53.72M
• there should be some memory used by pod/container not show inside container.
• total = 32.141G

cgroup-limit(32.241) - pod-used(32.141) = 100M then, there is only 100M extra memory left for all virt-launcher processes growing.

harvester has an option to resolve mem from guest-os-memory (undercommit).

• reserve default 100M, this setup cause cgroup-oom-killed many times
• now I reserve 256M, oom not seen for days. 256+247=500M, still worried about this number not enough.

so, how much extra memory is needed enough for kubevirt internal task usage?

[fanlix]

by #4560 , overcommitGuestOverhead: true help this case?

[simonyangcj]

@fanlix I think with overcommitGuestOverhead: false kubevirt will add additional memory to virt-launcher pod for these process. You can checkout code here for detail if !guestResourceSpec.OvercommitGuestOverhead . And IMHO when virt-launcher pod in burstable Qos group you vmi won't get oom kill but host might.

[fanlix]

So, additional 300M(calculated by the code) is added by overcommitGuestOverhead: false config. This is enough for normal kubevirt usage. But VM of heavy load cost more memory than expected. Causing host-cgroup-oom-kill.

Increasing additional cgroup memory limit, or decreasing vm.memory.guest may both avoid cgroup-oom.

If anyone want to dig into the calculating code, I'm glad to offer the virt-launcher process memory detail.