Tracking: introduce telemetry span into fluvio protocol

[sehz]

Open Telemetry approach allows developers to observe complex behavior among Microservices. To take advantage of this capability for Fluvio, we need to inject span from client to SC and back to SPU.

In order to accomplish this, we need to add arbitrary metadata to fluvio protocol.

• [ ] Add metadata to fluvio protocol header
• [ ] Enhance Client with trace span to SC and SPU
• [ ] Enhance SC to pass span to SPU
• [ ] Enhance SPU to pass span to other SPU
• [ ] Test with other observability tools such as Jaegar, Zipkin

Requirements:

• The telemetry metadata should be optional on demand. For example, --telemetry flag in the CLI to send telemetry span.

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[cooperbow]

Is there any relevant literature? Maybe I didn't read the original literature carefully and couldn't find relevant information. Thank you very much!!!!

[drc-infinyon]

@digikata can you comment on this when you have a minute. We have talked about it. This is a good prompt that has come up a few times recently.

[sehz]

Here is doc on [Span](https://opentelemetry.io/docs/concepts/signals/traces/#:~:text=A%20span%20represents%20a%20unit,ID%20(empty%20for%20root%20spans)

[digikata]

To elaborate a bit more: @cooperbow, there are multiple OpenTelemetry uses w/ fluvio. This is a fairly specific internal use case.

In general, here are some OpenTelemetry use cases not covered by this issue:

• Fluvio can already be used to receive OpenTelemetry as data (outside the scope of this issue: though we should write up more and maybe focus on describing some connectors to facilitate this and maybe a blog to demonstrate a setup)
• OpenTelemetry can be used w/ its standard utilities (e.g. sidecars) external to fluvio to collect log info etc (this is also outside the scope of this issue)
• Emitting opentelemetry metrics directly from fluvio components (also outside the scope of this issue)

For this specific issue: It's more about using the OpenTelemetry format (span messages) to help with inter service tracing. This issue of adding it the the protocol, would let the code directly carry trace/span messages in distributed communication between services, as well as a way for services identify span data and append events to spans between services.

Happy to discuss more here or in our discord.