KeyboardInterrupt support / SIGINT handler

[tkf]

trafficstars

Currently PyJulia does not support KeyboardInterrupt. That is to say, in long-running Python and Julia computation, there is no way to terminate a sub-computation by catching KeyboardInterrupt as done in normal Python programming.

Aside: Recommended way to cancel current input in REPL is to use IPython 7.0 or above. Ctrl-C would cancel the input without causing SIGINT.

This problem is previously mentioned in: https://github.com/JuliaPy/pyjulia/issues/189, https://github.com/JuliaPy/pyjulia/issues/185#issuecomment-418614274

What follows is a summary of my understanding:

When PyJulia is initialized, libjulia takes over all signal handling. I couldn't find a way to disable this behavior. Julia documentation mentions that "Julia requires a few signal to function property." so it probably would not have something like Py_InitializeEx to initialize libjulia without installing signal handlers anytime soon.

What would be more easily achievable is to let Julia translate SIGINT to InterruptException and then let PyCall to translate it to Python's KeyboardInterrupt. I have implemented it in https://github.com/JuliaPy/PyCall.jl/pull/574 but it introduced a bug which is hard to track. I reported it in Julia: https://github.com/JuliaLang/julia/issues/29498. Note that this strategy is not perfect because long-running pure-Python computation or I/O cannot respond to SIGINT. If Julia implements some kind of signal handling https://github.com/JuliaLang/julia/issues/14675 then maybe we can call PyErr_SetInterrupt (which does not need GIL) from it.

[Jasha10]

I have a way to raise KeyboardInterrupt in python with SIGINT. The idea is to combine the signal.pthread_sigmask and signal.signal functions from python's standard library.

# tmp.py
import signal
from time import sleep
from typing import Any

import julia

julia.Julia(sysimage="sys.so")

from julia import Main


def my_sig_handler(signalnum: int, frame: Any) -> Any:
    sig = signal.Signals(signalnum)
    print(f"\nGot {sig=}")
    raise KeyboardInterrupt


# unblock SIGINT and set up a handler
signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGINT])
signal.signal(signal.SIGINT, my_sig_handler)
# Other signals can be handled too, e.g. SIGUSR1:
signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGUSR1])
signal.signal(signal.SIGUSR1, my_sig_handler)

while True:
    print("sleeping")
    sleep(1)

Here is what usage looks like at the command-line:

$ python3 tmp.py
sleeping
sleeping
sleeping
^C
Got sig=<Signals.SIGINT: 2>
Traceback (most recent call last):
  File "/home/Jasha10/tmp.py", line 28, in <module>
    sleep(1)
  File "/home/Jasha10/tmp.py", line 16, in my_sig_handler
    raise KeyboardInterrupt
KeyboardInterrupt

[MilesCranmer]

Hi @tkf and @Jasha10, I was wondering if I could ask what the status of this issue is?

For context: I am in the process of porting the package "PySR" to PyJulia (https://github.com/MilesCranmer/PySR/pull/87) rather than the current strategy of executing a Julia script. Having a working KeyboardInterrupt is the last item I am trying to fix before I merge things - it is quite important for the package because there are long-running executions which need to be stopped by the user.

Are there any currently known workarounds I could look at using? Thanks! Best, Miles

[Jasha10]

Let me record my recent observations. Disclaimer: I am not an authority on this package.

Python signal-handling setup:

Suppose you have a python script that uses pyjulia (via e.g. from julia import Main). Sending a SIGINT signal to the program (by e.g. pressing CTRL-C) has a different effect depending on whether the following four lines are included in your python script:

import signal
def sig_handler_keyboard_interrupt(signalnum, frame): raise KeyboardInterrupt
signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGINT])
signal.signal(signal.SIGINT, sig_handler_keyboard_interrupt)

NOTE: For the above to have an effect, the calls to signal.pthread_sigmask and signal.signal MUST happen after the pyjulia setup, e.g. after the import statement from julia import Main.

Effects of the signal-handling setup

Your python script may do computations in julia (e.g. Main.sleep(1)) and it may do computations in python (e.g. import time; time.sleep(1)).

• Behavior of pyjulia if signal-handling is set up (see above):
  • when julia computation is running: sending SIGINT does not immediately interrupt the computation. KeyboardInterrupt() is raised in Python after the julia computation has finished.
  • when python computation is running: sending SIGINT immediately interrupts the computation and raises KeyboardInterrupt().
• Default behavior of pyjulia (without the signal-handling setup above):
  • when julia computation is running: sending SIGINT immediately interrupts the computation and raises KeyboardInterrupt().
  • when python computation is running: sending SIGINT does not interrupt the computation. Repeatedly sending SIGINT results in a fatal error and program exit.

Thus we have a "pick-your-poison" situation. I do not know whether there is a way to have one signal (SIGINT) interrupt both python computation and julia computation.

Additional notes

It seems that julia-side signal handling is not yet implemented in the standard library, so currently signal handling on the python side seems like the way to go. Power users may consider registering a different signal besides SIGINT, e.g. SIGUSR1.

signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGUSR1])
signal.signal(signal.SIGUSR1, sig_handler_keyboard_interrupt)

This way, SIGINT could be used to interrupt julia computation and SIGUSR1 could be used to interrupt python computation.