skywalking
skywalking copied to clipboard
apache
[Python] fork() support for grpc protocol
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [x] Feature or performance improvement
Requirement or improvement
- Please describe your requirements or improvement suggestions.
Currently the Python agent grpc protocol works in a threaded environment but not when fork()ing in a multiprocessing one. In theory the grpc libs can work after forking but maybe only if the lib is initialized after the fork or maybe not, depending on the source of information. In testing grpc may work for quick small scripts where maybe nothing was sent before the fork but fail completely on a real multiprocessing server implementation for example.
The fork hooks are in place in the Python agent (before fork, after in parent and after in child), but I have not been able to determine the proper procedures to make the protocol transition successfully to both a forked parent and child and keep working. Any input from someone with more experience with grpc in a multiprocessing environment would be welcome.
In theory the grpc libs can work after forking but maybe only if the lib is initialized after the fork or maybe not
A quick search shows that it works only when forking is done before any channels are established. https://github.com/grpc/grpc/blob/master/doc/fork_support.md#111
A quick search shows that it works only when forking is done before any channels are established. https://github.com/grpc/grpc/blob/master/doc/fork_support.md#111
I did see that file and it is not exactly as you say - 1.11 "This allows forking after creating a channel. However, the channel must not have issued any RPCs prior to the fork." - which is what I saw in testing more or less. So for example would it be possible to shut down the channel before fork and recreate in parent and child after? So far I have not been able to so requested comments here.
Hello, according to the latest doc (which wasn't updated to reflect the active gRPC channel support in 2018), I think this can be implemented now.
https://github.com/grpc/grpc/blob/master/doc/fork_support.md#111
Just to reference some potential problems on forking - https://github.com/grpc/grpc/issues/18075
We also need to use the new callback method https://docs.python.org/3/library/os.html#os.register_at_fork to enhance the pre-forking server behaviours for Python 3.7+ (Py3.6 should continue to manually add hooks to the server startup config)