PythonCall.jl
PythonCall.jl copied to clipboard
JuliaPy
Ctrl-C interrupts not working inside python execution
Affects: PythonCall / JuliaCall
Describe the bug Interrupting during python execution is not working properly (not even on one thread). Keeping ctrl+c pressed interrupts the execution, but the next python call segfaults julia. This applies to both, PythonCall and PyCall.
julia> using PyCall
julia> py"""
import time
for i in range(100):
time.sleep(1)"""
^C^C^C^C^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT
^C^CERROR: InterruptException:
Stacktrace:
[1] macro expansion
@ ~/.julia/packages/PyCall/1gn3u/src/exception.jl:108 [inlined]
[2] #117
@ ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:38 [inlined]
[3] disable_sigint
@ ./c.jl:473 [inlined]
[4] pyeval_(s::String, globals::PyDict{String, PyObject, true}, locals::PyDict{String, PyObject, true}, input_type::Int64, fname::String)
@ PyCall ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:37
[5] macro expansion
@ ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:230 [inlined]
[6] top-level scope
@ REPL[2]:1
julia> py"""
import time
for i in range(100):
time.sleep(1)"""
[473012] signal (11.1): Segmentation fault
in expression starting at REPL[2]:1
_PyInterpreterState_GET at /usr/local/src/conda/python-3.10.14/Include/internal/pycore_pystate.h:117 [inlined]
PyUnicode_DecodeFSDefaultAndSize at /usr/local/src/conda/python-3.10.14/Objects/unicodeobject.c:4095
Py_CompileStringExFlags at /usr/local/src/conda/python-3.10.14/Python/pythonrun.c:1390
macro expansion at /home/htc/bzfsikor/.julia/packages/PyCall/1gn3u/src/exception.jl:108 [inlined]
pyeval_ at /home/htc/bzfsikor/.julia/packages/PyCall/1gn3u/src/pyeval.jl:34
julia> using PythonCall
julia> @pyexec """
import time
for i in range(100):
time.sleep(1)"""
^C^C^C^C^CWARNING: Force throwing a SIGINT
^CERROR: InterruptException:
Stacktrace:
[1] PyObject_CallObject
@ ~/.julia/packages/PythonCall/S5MOg/src/C/pointers.jl:297 [inlined]
[2] macro expansion
@ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:132 [inlined]
[3] pycallargs(f::Py, args::Py)
@ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:212
[4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::@Kwargs{})
@ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:230
[5] pycall
@ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:220 [inlined]
[6] Py
@ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:339 [inlined]
[7] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::Tuple{})
@ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:1212
[8] top-level scope
@ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:1260
julia> @pyexec """
import time
for i in range(100):
time.sleep(1)"""
[472692] signal (11.1): Segmentation fault
in expression starting at REPL[2]:1
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.4/Include/internal/pycore_pystate.h:133 [inlined]
maybe_freelist_pop at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:1133 [inlined]
tuple_alloc at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:43 [inlined]
PyTuple_New at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:74
PyTuple_New at /home/htc/bzfsikor/.julia/packages/PythonCall/S5MOg/src/C/pointers.jl:297 [inlined]
pynulltuple at /home/htc/bzfsikor/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:824 [inlined]
Your system Please provide detailed information about your system:
- Linux, Julia 1.10.4, both PyCall and PythonCall on newest versions.
It's true that Python code is not currently interruptible when called from PythonCall. That requires some signal handling magic that I think isn't currently possible in Python and Julia.
In your example you threw SIGINT by holding down Ctrl-C, I'm not surprised this would crash things as it's generally intended as a last resort to stop a program.
What happens if you just press Ctrl-C once and wait?
I can reliably get a segfault on one Ctrl-C, which looks like this:
[ Info: Received message: nothing
^C
[82624] signal (2): Interrupt: 2
in expression starting at /Users/dkleinschmidt/work/REDACTED/demo.jl:22
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 3128814 (Pool: 3125853; Big: 2961); GC: 4
[82624] signal (11.2): Segmentation fault: 11
in expression starting at /Users/dkleinschmidt/work/REDACTED/demo.jl:22
_PyUnicode_FromId at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
_PyImport_GetModuleId at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
wait_for_thread_shutdown at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
Py_FinalizeEx at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
Py_FinalizeEx at /Users/dkleinschmidt/.julia/packages/PythonCall/flx5V/src/C/pointers.jl:303 [inlined]
#37 at /Users/dkleinschmidt/.julia/packages/PythonCall/flx5V/src/C/context.jl:194
unknown function (ip: 0x139e3444f)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
_atexit at ./initdefs.jl:428
jfptr__atexit_79411.3 at /Users/dkleinschmidt/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
ijl_atexit_hook at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/init.c:280
jl_exit_thread0_cb at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./signals-mach.c:464
Allocations: 3128814 (Pool: 3125853; Big: 2961); GC: 4
I can't share the exact code unfortunately, but there's no multi-threading happening in julia and the python code that is executing is a call to confluent_python.Consumer.poll() (which ultimately is a shim for a librdkafka C function call...). I do think that librdkafka is multithreaded though if that makes a difference...
This is with
- Julia ~~1.9.4~~ 1.10.4
- PythonCall 0.9.22
- confluent_kafka 2.5.0
- Python 3.10.13
- MacOS 14.5 (23F79) (Macbook Air M1)
