Processes started as user `root` can't be terminated

[frans-fuerst]

Not sure if this relates to issue 112 but since it doesn't depend on the user which starts a process and the issue is closed, I give it a try.

On Ubuntu 22:04 with OpenSSH 1:8.9p1-3ubuntu0.6 I can run ssh root@localhost -t sleep 20 and terminate the spawned process by pressing CTRL-C. So I guess despite I'm not root, ssh manages to forward the SIGINT to sleep the right way.

With asyncssh (2.14.2) for the regular user I get the same behavior:

async with asyncssh.connect("localhost", request_pty="force") as conn:
    async with conn.create_process("sleep 20") as process:
        await process.wait()

(As expected, without request_pty="force" terminating the process with CTRL-C does not work.)

However starting the same thing as root behaves as if I didn't supply request_pty="force":

async with asyncssh.connect("localhost", username="root", request_pty="force") as conn:
    async with conn.create_process("sleep 20") as process:
        await process.wait()

Spawned this way sleep can't be terminated anymore, neither by having SIGINT forwarded naturally nor by killing it explicitly.

I also tried to add term_type='ansi' to the create_process call (as suggested in the issue mentioned above) but without effect.

Expectation would be to achieve the same behavior as with command line ssh, providing -t for all provided users.

(I've posted a question on StackOverflow: https://stackoverflow.com/questions/77963100/how-to-kill-a-command-started-via-asyncssh-as-root)

[ronf]

I'm not sure I'm completely following what you're trying to do here.

If you run this code in a local Python interpreter and then hit Ctrl-C (assuming that is set as "intr" in your local TTY settings), that will trigger a KeyboardInterrupt exception in the local Python process. Since that's not being caught in the sample code, this will likely result in something like:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/private/tmp/t1.py", line 8, in run
    await process.wait()
  File "/private/tmp/asyncssh/process.py", line 1476, in wait
    await asyncio.wait_for(self.communicate(), timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
    return await fut
           ^^^^^^^^^
  File "/private/tmp/asyncssh/process.py", line 1365, in communicate
    await self.wait_closed()
  File "/private/tmp/asyncssh/process.py", line 1132, in wait_closed
    await self._chan.wait_closed()
  File "/private/tmp/asyncssh/channel.py", line 790, in wait_closed
    await self._close_event.wait()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/t1.py", line 10, in <module>
    asyncio.run(run())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 123, in run
    raise KeyboardInterrupt()
KeyboardInterrupt

It looks to me like the context managers in this case will cause the SSH connection to close cleanly despite the traceback here, and from what i can tell the remote ssh server looks like it might generate a SIGHUP in this case on the "sleep" running as root, and on my local system here that does cause the sleep to exit early, even when I'm logging in as "root".

Could you be doing something on your root user which is preventing the SIGHUP from making it through to the "sleep" process?

It may also vary from one SSH server to another (I was using OpenSSH here), and perhaps also could vary depending on the shell you are running for the user you are running as.

In any case, you should not expect Ctrl-C to be "forwarded". If you want that, you'd need to do something like the following, if you are running an SSH server new enough to properly handle signal requests:

async with asyncssh.connect("localhost", request_pty="force") as conn:
    async with conn.create_process("sleep 20") as process:
            try:
                await process.wait()
            except asyncio.CancelledError:
                process.send_signal('INT')

Interestingly, I still see the remote process exit with SIGHUP when I do this, even though logging shows AsyncSSH sending the SIGINT. I haven't dug very deeply into why that is yet, but it's sort of outside of AsyncSSH's control.

Regarding requesting a PTY, that really shouldn't matter here one way or another, as you aren't trying to write to stdin to trigger a signal on the remote system. That's the only time a PTY would matter, as discussed in #112.

[frans-fuerst]

Ok, it looks like I managed to create this problem by myself, having a command=<CMD> <PUBKEY> line in my /root/.ssh/authorized_keys file, which explains different handling of root/non-root connections.

Two remarks though, in case someone is stumbling upon this thread:

1. without knowing the exact circumstances for SIGINT being forwarded, it usually works this way. I just didn't handle KeyboardInterrupt in order to keep the snippet short. You can leave out the try/except and just wrap into suppress(KeyboardInterrupt) and you're fine:

async def main() -> None:
    async with connect("localhost", username="root", request_pty="force") as conn:
        async with conn.create_process("sleep 120") as process:
            await process.wait()

if __name__ == "__main__":
    with suppress(KeyboardInterrupt):
        asyncio.run(main())

2. More importantly request_pty="force" seems to be important. If you leave it out await process.wait() blocks even on CTRL-C and you won't even reach the except block.

Anyway, for me your hint Could you be doing something on your root user.. led me to the right spot and asyncssh is now working for me, thanks! (while I now wonder why ssh root@localhost -t ... worked, since it used the same mis-configured login..)

[ronf]

Glad to hear you got it working.

To expand a bit on the request_pty issue, AsyncSSH and regular command-line OpenSSH are very different about the handling of the local TTY (or whatever you have attached as stdin/stdout/stderr). With OpenSSH acting as a client, it's going to open a remote session and automatically forward all input on stdin to the remote system, and forward all output on stdout/stderr from the remote system back. AsyncSSH doesn't do anything like that, at least not by default. It's meant for you to programmatically drive the input & output, so it's up to you to decide what you want to send to the remote system, and what you want to do with the output that comes back.

Related to the above, OpenSSH will automatically set your local TTY into character-at-a-time mode and turn off any signal generation on the local machine when you request a remote PTY. Then, it will forward all your input (including the Ctrl-C) to the remote system as input. If you have requested a PTY on the remote system, this Ctrl-C will then be translated by the remote TTY as a SIGINT, assuming the default stty settings are in place. This will only happen if you request a PTY, though. Without one, the local terminal will still be processing those special characters, and the SIGINT will get generated and handled locally.

As discussed in #112, you can explicitly write a Ctrl-C with process.stdin.write('\x03') if you want to let the remote TTY generate a SIGINT for you, as long as you have requested a PTY. This would be an alternative to using send_signal(), and might be needed for older versions of OpenSSH server which didn't support those signal requests. You're not going to get this happening automatically when you hit Ctrl-C locally, though, as your local TTY's stdin/stdout/stderr is not automatically forwarded to the remote system the way it is with OpenSSH. Also, Python will capture the Ctrl-C and generate the KeyboardInterrupt, so you'd have to do something like put the write() call here into an "except" block. If you suppress the KeyboardInterrupt, AsyncSSH still won't pass anything through when you hit Ctrl-C locally. You need to do either the send_signal() explicitly or you need to request a PTY and write() whatever control character you want which is set to generate a signal on the remote TTY.

If you don't request a PTY, you'd be stuck with only being able to use send_signal(), which may not work on all SSH servers. Similarly, when not requesting a PTY on OpenSSH, any control characters you hit will be processed by the local TTY, and that will probably cause your SSH client to exit rather than passing the signal through. That may or may not trigger the remote process to clean up, depending on whether the SSH server generates something like SIGHUP and how the remote process handles that.