grpc
run_poller error when closing transport
What version of gRPC and what language are you using?
grpcio 1.54.0 grpcio-status 1.54.0
What operating system (Linux, Windows,...) and version?
Debian bullseye
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.11.2 gcc (Debian 10.2.1-6) 10.2.1 20210110
What did you do?
It's not deterministic but we're facing problem that we're getting
grpc_shutdown starts clean-up now
run_poller: UNKNOWN:Timer list shutdown {created_time:"2023-06-05T07:41:29.408344537+00:00"}
error when shutting down gRPC connection (it's probably coming from google-cloud-pubsub library but I'm not sure as we're integrated with couple of services that are using gRPC as a communication layer). We have a process where we're starting and shutting down gRPC connections pretty often (like once per 10s) and after some time we're getting this error on STDERR. I'm not sure what's the root cause of this issue. I can enable more debug logs on prod but not all as there is too many and this error is not showing up so often.
Additionally we're getting:
Error in sys.excepthook:
{}
Original exception was:
(yes, empty message that something went wrong with excepthook) but I'm not sure if those errors are connected.
What did you expect to see?
No errors on stderr
Did you make any progress on this case? I am facing the same issue when stopping and restarting gRPC stream clients.
We haven't seen the original error for days, but we're still seeing this:
Error in sys.excepthook:
{}
Original exception was:
We don't use sys.excepthook in our library. So this is likely coming from other part of your codebase. It looks like exception this is catching has been cut off from the log you shared in your last comment, could you share the full log with us.
The problem is that it's the whole output, displayed onto the stderr. It happens when we're finishing script, it looks like it's some sort of async teardown fired when deleting object. If it's not from your lib then I think we can close this issue.
This is definitely caused by grpc. it only happens when the poller thread is still alive at shutdown.
English is my second language, so sorry in advance.
I also got
Error in sys.excepthook: {} Original exception was:
when running tests for my service with pytest. Each test uses object that has grpc.aio.server. Tests are green, but in the end I get this message, one for each test
As a result, CI for my service is red, which is very inconvenient.
I tried to call at the end of test
await self.server.stop(self.cfg.grace_shutdown)
await self.server.wait_for_termination()
(which is also what real service is doing on shutdown), but the error persisted. When I comment out creation of grpc server, the error is gone. I also tried shutting down ThreadPoolExecutor that I pass to grpc.aio.server but to no avail.
I also noticed that awaiting for server.stop() when no server.start() was called just freezes indefinetly.
@justanotheruser Can you share a simple ( client.py server.py ) example with the issue.
More than 30 days have passed since label "disposition/requires reporter action" was added. Closing this issue. Please feel free to re-open/create a new issue if this is still relevant.
April 2025 and the issue is still alive and "well"... any hopes for looking into this?
April 2025 and the issue is still alive and "well"... any hopes for looking into this?
This solved the issue for me: https://github.com/grpc/grpc/issues/22365#issuecomment-2254278769