feat(telemetry): add custom headers for OTLP exports (metrics, tracing, access logs)

[codefromthecrypt]

What type of PR is this? feat(telemetry): add new feature

What this PR does / why we need it: Adds support for custom headers on all OTLP/gRPC exports (metrics, tracing, and access logs), enabling authentication with collectors like Elastic Cloud, Datadog, or cloud providers that require API keys or bearer tokens.

This uses Envoy's GrpcService.initial_metadata to send headers as gRPC metadata.

Release Notes: Yes

Notes: This PR includes an example showing authenticated OTLP exports for all three signals. The example uses otel-tui, but could use anything else.

Changes

• Metrics: Added Headers field to ProxyOpenTelemetrySink in telemetry.metrics.sinks[].openTelemetry
• Tracing: Added Headers field to TracingProvider in telemetry.tracing.provider
• Access Logs: Added Headers field to OpenTelemetryAccessLog in telemetry.accessLog.settings[].sinks[].openTelemetry

All three use the same pattern: a list of HTTPHeader objects with name and value fields.

FAQ

Why not use SecretObjectReference for sensitive headers?

For EnvoyProxy config that supports file-based standalone mode, Kubernetes secrets are not available. Plain text headers are needed regardless.

For example, Honeycomb requires non-secret headers like x-honeycomb-dataset. Secret support can be added in a follow-up for Kubernetes mode.

[codecov[bot]]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 72.41%. Comparing base (76d4076) to head (207b78d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7674   +/-   ##
=======================================
  Coverage   72.40%   72.41%           
=======================================
  Files         234      235    +1     
  Lines       34562    34586   +24     
=======================================
+ Hits        25026    25045   +19     
- Misses       7748     7753    +5     
  Partials     1788     1788           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codefromthecrypt]

ok made a revision also to PR desc to hopefully answer questions in comments. I will revisit this again monday if there is more feedback.

[codefromthecrypt]

pulling into draft while I remove the port dodging things for things we can't control yet.

[codefromthecrypt]

also I just realized we never added this for tracing either, so will add that to the PR

[codefromthecrypt]

ok I updated this as I noticed the same thing was missing everywhere, in logs metrics and tracing. fixed so that the tests are coherent and we don't need to go back and clean this up multiple times. There's been a history of fragmentation and not following up later, so doing otel holistically for one thing is a good thing.

[codefromthecrypt]

@arkodg @zirain I will mark this draft again and rebase after #7695 is merged because without those changes, the integration test for headers here won't pass.

[codefromthecrypt]

will undraft after #7697 because I noticed metrics are unreadable in otel-tui until that change (many otel backends require delta temporality)

[codefromthecrypt]

@arkodg @zirain I looked carefully at my screenshots and noticed that the logs didn't include any otel attributes. this is a bug, even if very small one. I raised https://github.com/envoyproxy/gateway/pull/7720 as we shouldn't merge this until it works like one would expect (the example should be valid and actually have attributes in otel for the log).

once that is merged I think we're finally good. thanks for your patience

[codefromthecrypt]

updated the screenshots based on the current branch which cherry-picks and shows the attributes for the log messages get to otel