gRPC: only first created channel is instrumented?
How do you use Sentry?
Sentry Saas (sentry.io)
Version
2.46.0
Steps to Reproduce
- Enable Sentry's
GRPCIntegration() - Create multiple gRPC channels with
channel = grpc.secure_channel(...)(e.g., for talking to different services) within a single application. - Do gRPC calls over these channels.
Expected Result
All gRPC calls are instrumented and show up in traces.
Actual Result
Spans are only created for some gRPC calls. Distributed tracing also does not work correctly, that is, the traces on the gRPC server are not connected with the gRPC client (in a Django app).
I suspect the problem is in the gRPC instumentation code:
https://github.com/getsentry/sentry-python/blob/6c6705a3d990559a80a48477a873d3171b928b12/sentry_sdk/integrations/grpc/init.py#L51-L58
The global variable ClientInterceptor._is_intercepted is used to do instrumentation only once. But this way only the first gRPC channel created will be instrumented: from the second secure_channel(...) call on, this variable is already true and instrumentation will be skipped.
I guess this is to avoid adding the interceptor twice when using grpc.secure_channel and grpc.intercept_channel? The integration does not use the ClientInterceptor._is_intercepted global variable for instrumenting grpc.aio.secure_channel after all.