opentelemetry-collector
opentelemetry-collector copied to clipboard
open-telemetry
ctx propagation in otlp/http vs otlp/grpc
Context I have written a custom auth extension which derives a key called consumer. I want to batch requests based on consumer by using the batch processor and use the same in a custom exporter. Issue 1: Batch processor uses the incoming metadata keys to batch. I can set the client.Metadata in my custom auth extension but the otlp/grpc receiver overrides client.Metadata. Issue 2: For the exporter, I tried setting client.Auth as well in my custom auth extension but the batch processor does not respect it. It creates a new context with only metadata, discarding the Auth (if set) by any authenticators.
Describe the bug I have created a custom server authenticator extension. At the end I am calling the following:
return client.NewContext(ctx, client.Info{
Metadata: client.NewMetadata(map[string][]string{
MetadataKeyId: {key},
}),
}), nil
Basically, I am trying to set some information in the metadata.
Now in one of my subsequent processors, I am trying to access this key. It works fine for http requests but the key is not present for grpc requests.
Steps to reproduce
What did you expect to see? Context should be preserved for both otlp/http and otlp/grpc
What did you see instead?
What version did you use? v0.99.0
What config did you use?
extensions:
customauthextension:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
include_metadata: true
max_recv_msg_size_mib: 16
auth:
authenticator: customauthextension
http:
cors:
allowed_origins:
- '*'
endpoint: 0.0.0.0:4318
include_metadata: true
auth:
authenticator: customauthextension
exporters:
logging:
processors:
customprocessor:
service:
extensions:
- customauthextension
pipelines:
logs:
receivers: [otlp]
processors: [customprocessor, batch]
exporters: [logging]
*** Additional Context *** I have been trying for a way to propagate context data from my custom authenticator to all processors and exporters for some time now. Here is a related PR regarding the same: https://github.com/open-telemetry/opentelemetry-collector/pull/10002
Update: If I set a custom context key in my authenticator like :
return context.WithValue(ctx, mycustomKey, key), nil
and then access it with
ctx.Value(mycustomKey)
in my processor, it worksss!!
I guess the root cause is that in grpc an interceptor to enrich the incoming context is called after the extension's auth method which is overwriting the Metadata set by the extension.
I need the batch processor to run after my customprocessor which requires a set of metadata keys to be set. However I am setting the metadata key in the auth extension which is being overidden in grpc requests.
@TylerHelmuth Amend the following line located at https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/configgrpc.go#L413:
- uInterceptors = append(uInterceptors, enhanceWithClientInformation(gss.IncludeMetadata))
+ uInterceptors = append(enhanceWithClientInformation(gss.IncludeMetadata), uInterceptors...)
This gives us control over playing with the Metadata in our interceptors.
@grandwizard28 , are you able to create a new test case here, demonstrating the problem? By looking at the code, I'd say that there's no override of these values, but perhaps I'm missing some information.
https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/configgrpc_test.go
Sample test here https://github.com/open-telemetry/opentelemetry-collector/pull/10162