apache
opendal python could hang while using with multiprocess
Is there a way to reproduce this issue? Maybe @TennyZhuang can share more details here?
I write some scripts to reproduce it, but failed, can you offer more detail for it?
This is related to https://github.com/tokio-rs/tokio/issues/4301.
The best we can do is return an error when Python users attempt to use fork.
This issue isn't limited to OpenDAL—it's relevant to all Python crates that depend on Tokio.
One such use case is for pre-forking servers running multiple worker processes. Many Python and Ruby webservers are pre-fork servers, e.g., Gunicorn, Puma for saving memory. A typical timeline for such servers:
- A web server runs as a master process, loading web application into memory.
- The master process forks itself, utilizing Copy-on-Write to save memory for child worker processes.
There is also a "refork" server which reaps worker processes and forks a warm worker process. When working with third-party libraries, web application developers have to releases resources during fork, e.g.,
- tokio resources
- locks and mutex
- file descriptors
Another question is if it's wrong to open an Operator while forking. A service's existing caches, listers and async tasks are useless to another process.
