Switch .NET tracer to injecting both base64 & binary headers

[veerbia]

Summary of changes

• Switch .NET tracer to injecting both base64 & binary headers
• Binary headers can be disabled via a config variable: DD_DATA_STREAMS_LEGACY_HEADERS=false (defaults to true)
• .NET tracer, when extracting the context from headers looks at both base64 & binary headers

Reason for change

Data Streams previously used binary encoding in Kafka headers. This was causing issues in cross language communication because of the difference in negative byte handling. That’s why we switched to base64 encoding. Today, .NET is the only remaining tracer using binary encoding for Kafka, SQS & RabbitMQ.

This is causing 3 issues:

• Communication between .NET & Java breaks, since Java doesn’t support negative bytes in Kafka headers
• Manual instrumentation breaks between .NET and any other language, since manual instrumentation uses base64 encoding
• .NET can’t consume payloads from any other tracers, since the other tracers encode in base64

Also, byte headers are causing a crash of the .NET application (not reproduced yet).

Implementation details

• Injects base64-encoded headers by default to ensure cross-language compatibility.
• Binary headers are injected only if DD_DATA_STREAMS_LEGACY_HEADERS (default: true) is enabled for backward compatibility.
• Prefers base64 headers for extraction. If base64 is unavailable or malformed, it falls back to binary headers when legacy headers are enabled.
• A new configuration setting DD_DATA_STREAMS_LEGACY_HEADERS controls whether binary headers are included.

Test coverage

I've added tests for the following cases:

• Inject_WhenLegacyHeadersDisabled_DoesNotIncludeBinaryHeader

  • Ensures that when legacy headers are disabled, the binary header is not included in the injected headers.

• Extract_WhenBothHeadersPresent_PrefersBase64Header

  • Confirms that when both Base64 and binary headers are present, the Base64 header is preferred for extraction.

• InjectedHeaders_HaveCorrectFormat

  • Validates that injected headers are in the correct format, especially checking if the Base64-encoded header is valid and correctly decoded.

• InjectHeaders_WhenLegacyHeadersDisabled_DoesNotIncludeLegacyHeader

  • Ensures that legacy headers are excluded when legacy header support is disabled.

• Inject_WhenLegacyHeadersEnabled_IncludesBothHeaders

  • Confirms that both Base64 and binary headers are injected when legacy header support is enabled.

• Extract_WhenBase64HeaderIsMalformed_ReturnsFallbackToBinary

  • Verifies that if the Base64 header is malformed, the system falls back to using the binary header for extraction.

Other details