Is PyCall thread safe?

[gsdean]

Seems that when we do anything with PyCall on multiple threads we get Segmentation faults.

Happy to provide more context, just wanted to check if this was designed to be in a multi-threaded environment first

[mrkn]

No, the current pycall isn't thread safe both Ruby and Python sides.

[ziaulrehman40]

Does this mean that if I queue say lot of background jobs in ruby, which can run concurrently obviously, those jobs cannot call pycall's functions at the same time? And i will have to reduce my concurrency for such jobs to 1 for pycall to work(which won't be practical for me, but confirming so i can decide my course of action)

[mrkn]

@ziaulrehman40 No, they can't call Python via pycall at the same time. It is better that you make the single thread from which calls Python.

[ziaulrehman40]

Well, that's a deal breaker for my use case. Most payloads are web related in today's world and require concurrency. I wonder what makes it fail in those scenarios and is there anyway community can help fix it.

For my use case, i had to use https://github.com/camelot-dev/camelot python library in my ruby code, we have decided now to just use its CLI option and call that CLI with system method of ruby. And as that CLI spits out files on disk, we can process them later. Giving us our concurrency. Obviously not best solution but solves our issue.

Thanks for the great effort in PyCall though, and I really hope we can enjoy this in concurrent scenarios soon.

[simonfranzen]

Any news on this topic? We want to call python libraries within a background job (sidekiq+rails).

[nhorton]

Is there any info on where the pycall gaps are on this so that others can contribute? Like @simonfranzen, we need to use the same stack.

[davidsbailey]

Any news on this topic? We want to call python libraries within a background job (sidekiq+rails).

@simonfranzen you could consider rails + resque, which uses a separate process for each job.

[jeremyhaile]

It definitely feels like this should be thread-safe, given how common threaded usage is these days, with web servers and background jobs.

But if it's not thread-safe, it also feels like the main README should have a big warning about that. We've spent a lot of time trying to figure out why our application is seg faulting!

Has anyone worked around this by wrapping pycall usage in a semaphore? (that's what we are trying now)

(we are also calling this from sidekiq background jobs)

[mrkn]

@jeremyhaile I want to accept your pull-request if you can make pycall thread safe even if it doesn’t introduce any overhead to single-threaded applications.

By the way, I’m working for streamlit-julia-call in my job nowadays. I’ve succeeded to bridge between multithreaded Python application and Julia in that project. I believe we can employ the similar approach in this project. When I have more time to tackle this issue again in the future, I want to try this approach. However, unfortunately, I am currently very busy and cannot afford to dedicate time to this project, so I hope someone eager to resolve this issue quickly can take over for me.