Add Support for generating / refreshing Auth Header in auto instrumentation

[DylanRussell]

Is your feature request related to a problem?

In order to authenticate to some Google APIs, we need the Authorization header to be added and then periodically refreshed in the OTLP exporter.

This is possibly by injecting ChannelCredentials (for grpc) or Session (for HTTP) into the OTLP exporters

Currently for auto instrumentation there's no way to inject these objects.

Describe the solution you'd like

Add new environment variables (OTEL_EXPORTER_OTLP_CREDENTIAL_PROVIDER, OTEL_EXPORTER_OTLP_LOGS_CREDENTIAL_PROVIDER, OTEL_EXPORTER_OTLP_METRICS_CREDENTIAL_PROVIDER, OTEL_EXPORTER_OTLP_TRACES_CREDENTIAL_PROVIDER) to the sdk. Alternatively add 2 new environment variables OTEL_EXPORTER_OTLP_GRPC_CREDENTIAL_PROVIDER for the GRPC OTLP exporters, and OTEL_EXPORTER_OTLP_HTTP_CREDENTIAL_PROVIDER for the HTTP OTLP exporters. These should actually be named OTEL_PYTHON_... according to https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#language-specific-environment-variables because they aren't part of the official environment variable spec.

This environment variable will be set to an entry point that when loaded and initialized returns a ChannelCredentials (for grpc) or Session (for HTTP) object, auto instrumentation code will then pass that into the OTLP exporter(s). The OTLP exporters already accept these objects in their constructors. We could load these in the OTLP exporter classes OR in the configurator class. The exporters currently don’t use the entry_points API at all, which is needed for this approach to work.

I prototyped this solution here.

ChannelCredentials and Session handle automatically setting and refreshing the Authorization header.

It’s already standard in the auto instrumentation code to use environment variables to configure the OTLP exporters, so I think this is the most straightforward option.

Describe alternatives you've considered

An alternative is to use the existing environment variables (OTEL_METRICS_EXPORTER, OTEL_TRACES_EXPORTER, OTEL_LOGS_EXPORTER) that inject metrics, traces, and log exporters. An initialized exporter with ChannelCredentials / Session could be passed to them. The code would have to be updated to accept class instance objects, instead of only Class objects.

This solution is fine, but it requires users to set multiple environment variables instead of only 1 in cases where they want to send multiple kinds of telemetry to google's API.

It also could be confusing for users that they need to specify an exporter that isn't the default OTLP exporter, when it is technically the default OTLP exporter that is being used under the hood.

Another alternative solution is to add a new environment variable (OTEL_EXPORTER_OTLP_CUSTOMIZER, and log/metric/trace equivalents) to the sdk. I prototyped this at https://github.com/open-telemetry/opentelemetry-python/pull/4452.

This environment variable will be set to an entry point that when loaded and initialized returns an ExporterCustomizer class which has a single configure_exporter function, that takes an OtlpExporter class, initializes it with whatever arguments it wants to (ex: passing in the Session or ChannelCredentials etc.), and returns the class instance.

This would allow vendors to customize the OTLP exporters for users instead of asking users to set 5 or 6 environment variables.

I'm fine with this approach too, but again it could be unclear to users what exactly the customizer is doing to their exporters.

Additional Context

How Java code does it: 1, 2.

Would you like to implement a fix?

Yes

[DylanRussell]

There's an open PR in go (https://github.com/open-telemetry/opentelemetry-go/pull/6362/files) and a merged PR for Java (1, 2) to support dynamic headers in the OTLP exporter. Meaning the exporter take function(s) that get invoked on each export call to produce header(s) that are added to the request.

I proposed something like that in https://github.com/open-telemetry/opentelemetry-python/pull/4431, but it was pointed out that Session and ChannelCredentials exist and can do what I need.

There's also an OTel Spec discussion around how to add authentication to HTTP OTLP exporters.

[DylanRussell]

TLDR:

We could create a customizer class to solve this -- see https://github.com/open-telemetry/opentelemetry-python/pull/4469 for what I mean by that.

Alternatively:

We do something like this.

Add a new environment variable (OTEL_EXPORTER_OTLP_CREDENTIAL_PROVIDER), which will be set to an entry point that when loaded and initialized returns a ChannelCredentials (for grpc) or Session (for HTTP) object. The Configurator class loads the entry point and passes it to the exporters.

Or we could take a slightly less generic approach and add 2 new environment variables (OTEL_EXPORTER_OTLP_GRPC_CREDENTIAL_PROVIDER, OTEL_EXPORTER_OTLP_HTTP_CREDENTIAL_PROVIDER), which are set to entry points that return a ChannelCredentials and Session object respectively. The OTLP exporter classes themselves can load these in.

[emdneto]

@DylanRussell Are other SDKs using the wording as credential providers or authenticators like in the collector?

[DylanRussell]

Hmm in Java I just see the word authentication a bunch (https://opentelemetry.io/docs/languages/java/sdk/#authentication, https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/gcp-auth-extension/README.md). I looked at a few other languages but didn't find any auth related wording at all. Anyway it seems reasonable to use that term instead of credential provider

[aabmass]

Are other SDKs using the wording as credential providers or authenticators like in the collector?

This was discussed in the Spec today. Seems like, yes, it would be useful to have this as a top level thing for exporters to use generically and we can probably model off how to collector does it

[DylanRussell]

How the collector does it seems very close to my OTEL_EXPORTER_OTLP_CREDENTIAL_PROVIDER approach.

The collector for client auth (auth on outgoing requests instead of incoming) has extensions provide a http.RoundTripper or credentials.PerRpcCredentials object, for example: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/oauth2clientauthextension/extension.go#L92-L113.

These are analogous to ChannelCredentials and requests.Session in python.

[srprash]

@DylanRussell Seems like Requests has a built-in mechanism to support custom authentication. It will probably be simpler and more specific to the use-case to have users provide a custom implementation of AuthBase than an entire custom Session object.

[DylanRussell]

Hmm what if we accept both AuthBase and requests.Session as objects that can be provided by the environment variable?

For talking to Google APIs we have a requests.Session object we want to pass in: https://google-auth.readthedocs.io/en/latest/user-guide.html#requests, but not an AuthBase object

[srprash]

what if we accept both AuthBase and requests.Session as objects that can be provided by the environment variable?

I'm ok with having both. Will need more environment variables though. @aabmass What do you think? Although a slight overkill, I can work with just the custom Session support for AWS auth if we decide not to support AuthBase.

[DylanRussell]

I think we could do it with a single environment variable by checking the type of the thing loaded in from the environment variable, check out the prototype for what I mean

[aabmass]

I think accepting just session is OK, since you can set the auth member on Session.

[aabmass]

@srprash correct me if I'm wrong, but I think the feedback from the Spec SIG was to try to generalize this so it can be generic across languages. Something similar to the global Propagator API which lets random components get the propagator and inject. Then exporters can do something like

class FooAuth(
  # Could be requests.AuthBase or grpc.Credentials or httpx.Auth depending on what
  # the exporter uses
  SomeClientAuthInterface,
):
  def authenticate(self, request) -> :
    auth_plugin = opentelemetry.sdk.auth.get_auth_plugin()
    auth_plugin.inject(request.headers)

class FooExporter(SpanExporter):
  def export(self, spans):
    # ...
    self.client.batch_write_spans(spans, auth=FooAuth())

Keep me honest though if I misunderstood @srprash