Need to apply a cgroup memory limit to user processes inside instances

[Saviq]

Run two snapcraft builds at the same time:

$ SNAPCRAFT_BUILD_ENVIRONMENT=multipass snapcraft
…

Notice that either will sometimes block for multiple seconds at a time. There's a chance our internal SFTP server is not concurrent enough…

[Saviq]

Sometimes this happens indefinitely, and the qemu process spins at 250% CPU here, but Ctrl+C on the snapcraft command makes it exit cleanly and everything unblocks. So I'm suspecting our mount going crazy.

[popey]

Yup, I have been seeing this on my laptop. Snapcraft runs block for long periods and are very slow. I'm limiting the RAM and CPU assigned to each machine to ensure I still have some cycles for Slack and Firefox.

[Saviq]

What I did is multipass shell into the instance while it was still responsive and waited for it to get stuck... When it did, it basically ran out of memory, kswapd0 was hogging the CPU. At first I thought we were swapping out to disk, but realized there's no swap at all, which may be what the reason for this is. I remember someone telling me a while ago that having a tiny swap is basically required, otherwise the kernel goes around in circles, screaming, when it runs out of memory.

I'll be testing this solution out next week, another approach is running all the multipass exec commands in a cgroup that limits CPU and memory usage to 95% or so, leaving the rest to the control channel and the kernel to survive.

[townsend2010]

As mentioned on irc, let's seeing about a small swap and setting up cloud-init to add it to the instance.

[Saviq]

Unfortunately adding swap or limiting the amount of memory on the kernel command line, or upgrading the kernel to hwe, didn't help. The last thing we'll try will be the cgroup and limiting the amount of memory the multipass exec commands can allocate.

[Saviq]

At least 300MB of memory needs to be reserved for the system, in the case of a 2GB machine I applied this cgroup:

# /etc/cgconfig.conf
group memlimit {
  perm {
    admin {
      uid = root;
      gid = root;
    }
    task {
      uid = 1000;
      gid = 1000;
    }
  }
  memory {
    memory.limit_in_bytes = "1500M";
  }
}

Using cgexec -g memory:memlimit/ --sticky. On top of that, sysctl's vm.oom_kill_allocating_task=1. It took a couple minutes, but the compiler processes running out of memory were being killed:

[ 4695.743968] Memory cgroup out of memory: Kill process 31196 (cc1plus) score 247 or sacrifice child
[ 4695.864834] Killed process 31196 (cc1plus) total-vm:465032kB, anon-rss:389696kB, file-rss:588kB

Most importantly, the machine was responsive throughout. So the cgroup approach seems valid. We should forward the OOM kernel messages to the user in a warning or so.