JuliaPy
What does it take for pyjulia to drop GIL?
Numba has a nogil annotation, I'm wondering if we can somehow drop the GIL and what would that imply.
I think we would need to reformulate the binding as a C or Rust based extension rather than using ctypes
We do use CDLL for unix like systems: https://github.com/JuliaPy/pyjulia/blob/69b734f117e489904a07d20f46c8f96b56577779/src/julia/find_libpython.py#L83
If I understand correctly, it’s the one for libjulia that matters. https://github.com/JuliaPy/pyjulia/blob/69b734f117e489904a07d20f46c8f96b56577779/src/julia/libjulia.py#L220
yeah, I think because you're calling Julia via the libjulia, so whether or not GIL is dropped during call depends on how you loaded libjulia?
maybe it's worth getting perspective from @stevengj and @MilesCranmer ?
- Can we by default drop GIL (i.e. use CDLL)?
- Does it actually drop GIL and speed things up?
- Does it create corretness issues?
- Given 3, how can we drop GIL call by call
I would love to drop GIL. This would be great so that I can do other things in Python (like printing updates from another thread) while some large Julia processing is happening in the background.
Regarding 3, I'm not sure what correctness issues it would cause, if any. It's unclear to me if you'd be able to mutate any Julia variables while processing is already taking place, so maybe it's safe as-is...? Maybe we should just try it out and see.
yeah I'd like to try it, but I would prefer if we can present it as an option. In this case, it would be better if it can be a per-call site option, but a per-session option would do.
One case we know definitely won't work is if Julia needs to call Python stuff as call back, or mutate Python variable, but I feel like that's minor usage
I assume just having two libjulia handle should allow having this as a per call option with the ctypes api.
I can't quite figure out how to have a local testable pyjulia, clone and pip install -e . didn't quite work, or maybe I'm looking at the wrong place to print debug
I guess for one I can't figure out which handle is used when I do
import julia
#vs.
from julia import Base
I can't get any print statement to work -- these two must share the same Julia handle right? but which one / where I can't quite figure out
I think the main use case is when calling Julia code from Python, releasing the GIL would be useful.