IO from python-spawned threads causes deadlocks

[lassepe]

Description

Carrying over from #539

The following code reaches a deadlock:

from concurrent.futures import ThreadPoolExecutor, as_completed
from juliacall import Main as jl

jl.seval(
    """
function my_sqr()
    println("Julia get's stuck at this print statement issued from thread $(Threads.threadid())")
    a = rand(10)
    for ii in 1:size(a, 1)
        a[ii] = a[ii]^2
    end
    return sum(a)
end
"""
)

pool = ThreadPoolExecutor(2)

fs = {pool.submit(jl.my_sqr): ix for ix in range(10)}
for future in as_completed(fs):
    print("running")
    rank = fs[future]
    results = future.result()
    print("done")
    print(results)

Potential Fix

Adding ccall(:jl_enter_threaded_region, Cvoid, ()) as suggested in https://github.com/JuliaPy/PythonCall.jl/issues/539#issuecomment-2295200264 fixes this issue but not the original one of #539. PythonCall may consider to invoke jl_enter_threaded_region behind the scenes.

[cjdoris]

Indeed the semantics of how Julia and Python threads interact have not been totally worked out yet. In particular, Julia has a task scheduler which doesn't really like running on foreign threads. I think there are functions to let Julia adopt Python-created threads but I've not tried them out.

However for me, the advice in this section of the docs successfully makes the example work. I also needed to do pool.submit(jl.my_sqr._jl_call_nogil) to release the GIL while calling my_sqr - without it, the GIL is held by the Julia thread that is deadlocked, preventing the main Python thread from doing anything.

I would of course like to avoid having to explicitly do this yielding loop to make the threads run Julia code properly. Thanks for pointing out jl_enter_threaded_region. Hopefully some combination of this and thread adoption will make this all work.