systemd-oomd kills all processes inside a container

[thiblahute]

Describe the bug

I am using toolbox on silverblue 34 all day to do development inside vscode using toolbox-vscode and the container silently stops and I have to reload the vscode window. I think the issue is totally unrelated to vscode but it is just the way I am running my toolbox.

It looks like the command line of the main toolbox process inside the podman container is toolbox --verboseinit-container --home /var/home/thiblahute --home-link --media-link --mnt-link --monitor-host --shell /bin/bash --uid 1000 --user thiblahute and I am using zsh as default shell inside the container, I have the impression this process stop which leads to the container stopping but I have no proof of that at this point.

Steps how to reproduce the behaviour

1. Use toolbox for a while
2. See that the podman container stops randomly

Expected behaviour I can use toolbox forever without needing to restart it

Actual behaviour The toolbox container stops after a while, a few times a day and I have to restart it.

Output of toolbox --version (v0.0.90+)

toolbox version 0.0.99.1

Toolbox package info (rpm -q toolbox)

toolbox-0.0.99.1-1.fc34.x86_64

Output of podman version

Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.16.3
Built:        Mon Jun 14 21:12:29 2021
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)

podman-3.2.1-1.fc34.x86_64

Info about your OS

Fedora Silverblue 34

[thiblahute]

Any hints on that one? This issue is pretty annoying.

[HarryMichal]

Hi @thiblahute! Does the issue occur outside of VSCode? If not, then I'd suggest you to report this in owtaylor/toolbox-vscode instead. I personally have no idea how the scripts work.

@owtaylor, should we move this?

[thiblahute]

Hi @HarryMichal, good question, I just installed vscode inside the container and am going to use it today and see if it also reproduces without that script.

[Ayush1325]

Hi @HarryMichal, I can confirm that this happens outside vscode too. I am using toolbox to do KDE development and toolbox silently exists. I am not using any GUI, just a terminal. Can confirm this happens in both bash and zsh.

The exit always happens in the middle of compilation, so it might be related to file io or cpu utilization.

[thiblahute]

Installing vscode inside the container and working from there lead to other issues so I could not finish that experiment. I am having the container stop randomly and it happens basically at any time, I couldn't find any pattern myself.

[rlac]

I've had this happen a few times too, and it's always been systemd-oomd killing a toolbox when memory gets low. systemd-oomd operates on cgroups, and all processes running inside of each toolbox are in the same cgroup - so the toolbox as a whole is treated as a single process to be killed to reclaim memory.

[thiblahute]

I've had this happen a few times too, and it's always been systemd-oomd killing a toolbox when memory gets low. systemd-oomd operates on cgroups, and all processes running inside of each toolbox are in the same cgroup - so the toolbox as a whole is treated as a single process to be killed to reclaim memory.

That might totally be it in my case, killing toolbox as a whole sounds very suboptimal!

[debarshiray]

I wonder if we can somehow prevent the entry point (ie., the toolbox init-container command) from getting killed. The absence of that process makes the container look stopped.

[HarryMichal]

Or maybe adjust the cgroup namespace? We currently don't touch it which means each toolbx has its own private cgroup (according to Podman's documentation). We could add --cgroupns=host to the arguments used to create a toolbx container.

[debarshiray]

We could add --cgroupns=host to the arguments used to create a toolbx container.

Are you sure that --cgroupns=host will help with this?

[debarshiray]

I've had this happen a few times too, and it's always been systemd-oomd killing a toolbox when memory gets low. systemd-oomd operates on cgroups, and all processes running inside of each toolbox are in the same cgroup - so the toolbox as a whole is treated as a single process to be killed to reclaim memory.

Yes, you are right. @halfline pointed out the same thing over at https://github.com/containers/toolbox/pull/1207

GNOME Terminal puts each VteTerminal into a separate scope. I wish we could do something like that here.

[debarshiray]

We could add --cgroupns=host to the arguments used to create a toolbx container.

This is already the case for containers created with Toolbx >= 0.0.99.4 but it doesn't change the fact that all the processes running inside a Podman container share the same cgroup.