podman run/exec should not ignore the SIGHUP signal

[1player]

Adapted from discussion at https://github.com/containers/podman/discussions/19944

podman run and podman exec do not terminate their child processes when a terminal window is closed. Normally, when that happens, the kernel sends a SIGHUP signal to the foreground process, signal which is completely ignored by podman exec so they never have a chance to terminate and keep running, even if their pty is no longer "visible".

How to reproduce

• podman run -it --rm alpine
• Inside the container: sleep infinity
• Close the terminal window
• Notice how sleep infinity is still running

How it works normally

• Open a host shell, run sleep infinity
• Close the terminal window
• Notice that sleep infinity has been terminated, because the SIGHUP was able to reach it. This is the correct POSIX behaviour

---

This issue affects toolbox and distrobox containers, as long running programs are never terminated when a terminal window is closed. Whether this is intended behaviour, I would suggest adding a command-line option to podman to respect and passthrough SIGHUP signals so it is able to behave like a regular shell for these use cases.

See also: https://github.com/89luca89/distrobox/issues/966

[mheon]

Assuming the sig-proxy flag is set to true (the default), we should be forwarding that SIGHUP into the container when Podman receives it. As such, it's really down to what is running in the container to respond . sleep ignores most signals (we had to remove any use of it in our tests because of this; it ignores the normal SIGTERM we send to stop containers), so it continuing to run is unsurprising. I don't think we can easily change this behavior, either (required for Docker compat).

[1player]

Sorry, I don't think that's the case. I just tried the exact same reproduction with top instead of sleep, and it is never killed when the terminal is closed. And I doubt that sleep ignores SIGHUP because, as shown above, the process is killed by it in a normal shell, the issue is only with podman.

Any more information on this sig-proxy flag? I'm not sure what you're referring to.

[mheon]

Per the docs: Proxy received signals to the container process (non-TTY mode only). SIGCHLD, SIGSTOP, and SIGKILL are not proxied.

Which is actually incorrect - we definitely proxy in TTY mode as well. And SIGURG is not proxied because of Golang internals using it extensively. Oops.

You can see it in action by starting a container (I used podman run -t -i --rm fedora sleep 5000) and then hitting Podman with a signal (pidof podman and kill -HUP that PID). Which does confirm your theory that sleep dies to a SIGHUP, so something else must be going on. Regardless, the idea is that all signals received by a podman run or podman attach are forwarded to the container instead of being handled by Podman itself, so if the terminal is killing Podman, we're ignoring that and forwarding the signal on to the container, then exiting once the container exits.

[1player]

Sorry @mheon I cannot reproduce here: if I run podman run -t -i --rm fedora sleep 5000 and I kill -HUP that process, nothing happens. In fact, podman becomes irresponsive to Ctrl-C and requires a SIGKILL to terminate.

BTW I'm on podman-4.6.2-3.fc38.x86_64

[Luap99]

Sleep is ignoring these signals when you run sleep in a container as pid 1, this is becuase the default signal action is ignore as set by linux, see kill(2):

The only signals that can be sent to process ID 1, the init process, are those for which init has explicitly installed signal handlers. This is done to assure the system is not brought down accidentally.

If I run podman run -it alpine top sending SIGHUB works correctly because top will exit on that signal.

However I don't see any proxy code for podman exec so that would be the main issue.

[1player]

Fine, yes, the SIGHUP works with top. But it still does not solve the original problem I opened this issue for: if you run podman run -it alpine top and close the terminal, top doesn't get killed.

The rules which determine whether the kernel sends a process SIGHUP when a pty is closed are quite complicated, the point is of this issue is that process running under podman do not get killed as expected, and as such, might leave child processes laying about in running state because they have not been terminated properly.

If we have established podman passes through SIGHUPs just fine, we need to establish why no SIGHUP is sent to podman when the terminal is closed.

[Luap99]

we need to establish why no SIGHUP is sent to podman when the terminal is closed.

That would be a problem with your terminal, it woks fine for me with the default gnome terminal.

[1player]

EDIT: disregard. Affects both VTE-based terminals and Konsole.

Did a quick test, could not reproduce with Black Box (a VTE based terminal emulator from Flathub), but the issue exists with KDE's Konsole, which I feel is worth investigating, being the second largest desktop environment, and a behaviour that does only present with podman, and not a normal shell.

So the question is, what is podman doing, when running in Konsole, that causes it to misbehave from a regular shell, or a VTE terminal?

Also, given the linked issues of other people having the same problem in toolbox, is everybody using Konsole, and no GNOME user is affected?

On Wed, 4 Oct 2023, at 10:53, Paul Holzinger wrote:

we need to establish why no SIGHUP is sent to podman when the terminal is closed.

That would be a problem with your terminal, it woks fine for me with the default gnome terminal.

— Reply to this email directly, view it on GitHub https://github.com/containers/toolbox/issues/1400, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFIPSA55YSMSC5PVGMVS7DX5UWZ7AVCNFSM6AAAAAA43QGWNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBWGUZTAMJUHE. You are receiving this because you authored the thread.Message ID: @.***>

[1player]

Disregard the previous message. The issue affects both VTE and Konsole, so I wonder which terminal emulator are you using, @Luap99

There are three distinct behaviour going on here:

1. Regular shell. A command, such as sleep infinity, terminates with SIGHUP. Despite the top comment from @mheon, all sleep processes run under /bin/sh and /bin/zsh on my machine terminate upon receiving that signal. This is the correct behaviour.

2. podman run -it archlinux sleep infinity. Here podman is ignoring SIGHUP, despite the claims that it does not. Sending SIGHUP to the sleep process directly does not do anything either. This is not the expected behaviour.

3. podman run -it archlinux to spawn a shell, then run sleep infinity. In this case, podman does terminate SIGHUP, but does not kill the child sleep process. This is not the expected behaviour either.

In case number 3, the problem is actually hairier: podman itself and the sleep process are connected to two different ptys:

sph      2184083  0.4  0.0 1869704 47768 pts/5   Sl+  22:57   0:00 podman run -it archlinux
sph      2184289  0.0  0.0   3012  2080  pts/0    S+   22:57   0:00 sleep infinity

Note podman attached to pts/5 and sleep to pts/0.

Upon closing a terminal window, the kernel will send SIGHUP to the shell, which will forward it to podman and will terminate. But the sleep process, connected to another pty, won't be killed by the kernel. Again, this is not the expected behaviour. I expect podman to terminate all child processes when it terminates itself.

On a regular (sub)shell, it's quite simple: everything is attached to the same pty, so child processes get killed when the terminal closes.

[Luap99]

2. podman run -it archlinux sleep infinity. Here podman is ignoring SIGHUP, despite the claims that it does not. Sending SIGHUP to the sleep process directly does not do anything either. This is not the expected behaviour.

Again, yes it is expected because sleep is pid 1 and does not have signal handlers installed to explicitly handle SIGHUP. The default signal action for pid 1 is ignore as written above. The podman run command simply forwards all signals that it can if sleep is ignoring that signal then there is not much podman can do about that. What you can do is to start the container with --init then it will work correctly: podman run -it --init archlinux sleep infinity

3. podman run -it archlinux to spawn a shell, then run sleep infinity. In this case, podman does terminate SIGHUP, but does not kill the child sleep process. This is not the expected behaviour either.

This is not what I am seeing and technically not possible, once the container pid 1 process exits all child processes in this pid namespace are killed.

---

Upon closing a terminal window, the kernel will send SIGHUP to the shell, which will forward it to podman and will terminate. But the sleep process, connected to another pty, won't be killed by the kernel. Again, this is not the expected behaviour. I expect podman to terminate all child processes when it terminates itself.

I am using the stock gnome-terminal. To me this sounds like you terminal is starting to kill the processes (likely after trying SIGHUP first) so the podman run command will be gone. If I do this with gnome-terminal the podman run command was not killed which is expected as it ignores SIGHUP and just forwards that to the container process.

On a regular (sub)shell, it's quite simple: everything is attached to the same pty, so child processes get killed when the terminal closes.

Well a container is not a regular process and is not attached to the same tty for multiple reasons. A user can disconnect from the container at any time or start it in the background but also security as the contianer should not get control over your tty. Many containers run long running services and should not be killed when you close the terminal. And keep in mind these processes do not get killed (SIGKILL), they get a SIGHUP signal so they can choose to ignore that signal.

[1player]

I feel we are talking past each other, and claiming the issue I, and other users seem to have, which affects toolbox and distrobox, have does not exist.

The question is simple. How do I run a command in podman that terminates when the terminal window closes, like running a program under a shell does?

This is necessary to make toolbox and distrobox usage patterns viable. While as of today I routinely find plenty of processes that have not terminated because they have been started within a container and never killed as I expected when I closed the terminal window, especially the shell process.

~ % ps aux | grep 'zsh -l'
sph         4010  0.0  0.0  10864  7036 pts/0    Ss   Oct03   0:00 /usr/bin/zsh -l
sph         8195  0.0  0.0  10976  6944 pts/2    Ss+  Oct03   0:00 /usr/bin/zsh -l
sph        19417  0.0  0.0  23832 19812 ?        Ss+  Oct03   0:00 /usr/bin/zsh -l
sph        84531  0.0  0.0  10416  6852 pts/3    Ss+  Oct03   0:00 /usr/bin/zsh -l
sph       154035  0.0  0.0  10804  7108 pts/4    Ss+  Oct04   0:00 /usr/bin/zsh -l
sph       427381  0.0  0.0  10424  6820 ?        Ss+  12:05   0:00 /usr/bin/zsh -l
sph       441797  0.0  0.0 222544  2400 pts/4    S+   12:10   0:00 grep --color=auto zsh -l
sph       832440  0.0  0.0  10296  6604 ?        Ss+  Oct05   0:00 /usr/bin/zsh -l
sph      1077068  0.0  0.0  10104  6336 ?        Ss+  Oct05   0:00 /usr/bin/zsh -l
sph      1100861  0.0  0.0  10412  6928 ?        Ss+  Oct05   0:00 /usr/bin/zsh -l
sph      1157104  0.0  0.0  10104  6448 ?        Ss+  Oct05   0:00 /usr/bin/zsh -l
sph      2139456  0.0  0.0  10292  6896 ?        Ss+  Oct05   0:00 /usr/bin/zsh -l

This is what my Fedora Silverblue workstation, where most work is done in containers, looks like after three days of uptime.

Again, what incantation do I need to do for podman commands to terminate when a terminal window is closed?

On Fri, 6 Oct 2023, at 10:01, Paul Holzinger wrote:

2. podman run -it archlinux sleep infinity. Here podman is ignoring SIGHUP, despite the claims that it does not. Sending SIGHUP to the sleep process directly does not do anything either. This is not the expected behaviour.

Again, yes it is expected because sleep is pid 1 and does not have signal handlers installed to explicitly handle SIGHUP. The default signal action for pid 1 is ignore as written above. The podman run command simply forwards all signals that it can if sleep is ignoring that signal then there is not much podman can do about that. What you can do is to start the container with --init then it will work correctly: podman run -it --init archlinux sleep infinity

3. podman run -it archlinux to spawn a shell, then run sleep infinity. In this case, podman does terminate SIGHUP, but does not kill the child sleep process. This is not the expected behaviour either.

This is not what I am seeing and technically not possible, once the container pid 1 process exits all child processes in this pid namespace are killed.

Upon closing a terminal window, the kernel will send SIGHUP to the shell, which will forward it to podman and will terminate. But the sleep process, connected to another pty, won't be killed by the kernel. Again, this is not the expected behaviour. I expect podman to terminate all child processes when it terminates itself.

I am using the stock gnome-terminal. To me this sounds like you terminal is starting to kill the processes (likely after trying SIGHUP first) so the podman run command will be gone. If I do this with gnome-terminal the podman run command was not killed which is expected as it ignores SIGHUP and just forwards that to the container process.

On a regular (sub)shell, it's quite simple: everything is attached to the same pty, so child processes get killed when the terminal closes.

Well a container is not a regular process and is not attached to the same tty for multiple reasons. A user can disconnect from the container at any time or start it in the background but also security as the contianer should not get control over your tty. Many containers run long running services and should not be killed when you close the terminal. And keep in mind these processes do not get killed (SIGKILL), they get a SIGHUP signal so they can choose to ignore that signal.

— Reply to this email directly, view it on GitHub https://github.com/containers/toolbox/issues/1400, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFIPSDY67HXCLZ4ZRKXSA3X57CINAVCNFSM6AAAAAA43QGWNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQGIZTMOJXHA. You are receiving this because you authored the thread.Message ID: @.***>

[mheon]

I see no evidence that this is a general issue. Podman correctly forwards signals via --sig-proxy, which will result in application termination on terminal close (assuming the application properly responds to signals). I've tested this myself, we have tests in CI to verify this, and if this was a widespread problem we would have heard about it before. As such, I'm definitely leaning towards an environment problem of some sort on your end. Though, as @Luap99 mentioned, this is currently exclusive to podman run - podman exec will allow exec sessions to escape supervision as we don't have signal proxying there. Or even a way to kill exec sessions other than taking the entire container down. That'll need to be fixed.

The fact that you can have a podman run, send a signal to Podman, and it does not make it into the container is very unusual, and indicates that --sig-proxy is somehow broken in your setup. If you run a podman run with --log-level=debug and kill -HUP the Podman process do you see logs for signals being forwarded (Sending signal 1 to container $CID or similar)? Maybe try podman kill as well - the mechanism that uses to signal the container is identical to signal proxying. This code is relatively simple on our end (catch all valid signals, if caught send identical signal to container) so I don't know where bugs could happen. Maybe a Golang signal-handling issue, but that seems like a longshot?

This is also reminding me that Linux has special rules for signal handlers and PID 1 (PID1 cannot be affected by signals it did not explicitly register a handler for, including PID1 in a PID namespace - IE container), which could mean that some processes that are well-behaved outside containers can act strangely inside them with regard to signals. However, you mentioned earlier that top also was not stopped by proxied signals, and that definitely registers its own handlers - I can confirm easily that it dies to SIGHUP even inside a container.

[rhatdan]

One thing you could test is to enter the container via podman exec and then send the hup signal from inside, if the pid1 reacts then you know that podman is not forwarding the signal from the outside.

[1player]

Whatever the reason, podman does not operate as a shell, I can't keep stressing this enough, and for this reason it makes its operation under Fedora Silverblue and other toolbox-centred distros a pain in the butt.

I don't know if I'm not making myself clear, if I'm hallucinating, but it really takes a second of your time to replicate, and I'm not sure anyone has tried yet.

I'm on a fresh Fedora Silverblue in GNOME, if I open gnome-terminal and run sleep infinity, close the window, open a new one, the sleep process has been killed.

Conversely, if I run podman run -it archlinux sleep infinity, close the terminal, reopen it, both the podman process and the sleep process are still running, and in fact can only be terminated by SIGKILL, as if they're stuck.

Please take 15 seconds of your time to replicate this elementary example, see if I'm hallucinating, and please suggest how to tell podman to behave exactly like a regular shell. SIGHUP, PID 1 semantics, are all red herrings. I just want to tell podman to behave like a shell does, or to know that it is not possible so I should bark up a different tree. Right now I'm only told what I'm seeing is wrong.

[Luap99]

Conversely, if I run podman run -it archlinux sleep infinity, close the terminal, reopen it, both the podman process and the sleep process are still running, and in fact can only be terminated by SIGKILL, as if they're stuck.

As we said multiple times now yes this is normal and expected, you need to make sure your init process has signal handlers for SIGHUP and stops on it. So either use a different process who does that or use --init to achieve this. You can replicate this by running unshare -Up --fork sleep inf and see that the sleep process now ignores the SIGHUP singal because it will be pid in its namespace.

I just want to tell podman to behave like a shell does, or to know that it is not possible so I should bark up a different tree

Podman containers are not a shell and it does not behave like one, the container process will always be double forked into the background as containers are by design not attached to your terminal. You can have multiple podman attach for example.

For podman exec there is no solution as there is no sig proxy, that is something that needs to be looked at but podman run works like it should due the sig-proxy .

[1player]

OK I understand now. To return to my original problem seen with toolbox/distrobox, basically they run podman exec --interactive sh. It seems I would have to explicitely trap SIGHUP in the shell script (https://stackoverflow.com/a/60537655) to then manually kill children and clean up, is that correct?

Does this also apply in containers that share the PID namespace with the host, such as toolbox? Because inside of them, I see the host systemd running as PID 1, but I seem to understand the SIGHUP handling would have to be done by the process that podman execs directly.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[1player]

This is still an issue and I'd like to know if the assessment of this behaviour in my previous comment is correct, so I know where to start addressing it.

[rhatdan]

At this point it seems more like you have a question for toolbox then for podman. Podman should handle correctly the case where the primary process of the container exits, it should clean up the container, If toolbox executes a script that does not exit, then there is nothing Podman can do.

[1player]

@rhatdan @Luap99 I ask to reconsider this issue.

Here's what seems to be the problem. The problem is that processes started under podman ultimately have conmon as parent PID, and conmon itself is not associated to any TTY.

Here is top launched on the host machine:

~ % ps -def | grep top
sph        16526   16486  1 10:36 pts/6    00:00:00 top

Notice that it is attached to pts/6. Here's top launched in a container:

~ % ps -def | grep top
sph        17000   16958  3 10:37 ?        00:00:00 top

It is not attached to any TTY at all. So, closing a terminal emulator will not send any SIGHUP, so that second top process will keep running indefinitely until the machine is rebooted.

So the issue is not that podman run/exec ignore SIGHUP. The issue is that processes started in a container are unable to receive any SIGHUP, as they are not associated with any TTY, so will not be terminated if they are spawned in a terminal emulator, causing a constant resource leak.

What is the appropriate solution to this issue? Given that Fedora Atomic workflow is centred around container usage (spawned from your terminal emulator of choice), this is a real concern. It is not normal that I have to reboot my machine every day to clean the 30+ shell and conmon processes lying around consuming resources.

[rhatdan]

If I run a container with the terminal attached, when the terminal exits podman exits.

$ podman run -ti alpine top

Kill the terminal

The contianer will exit. As I understand it, the container does not exit, if you do not connect a terminal to it or run the container in -d mode.