[DRAFT] custom: Go plugin support for custom plugins to support OTel extensions

[robskillington]

Opening this for feedback on implementation. Have this working end-to-end.

Since OpenTelemetry extensions do not require telemetry processing (unlike plugins/extensions for input/output/filter/processing/etc that do require it in FluentBit) they are relatively simple to support since they just need to be "started" and "stopped" at process boot and config hot reloads.

One primary extension that tracing end users frequently require is the ability to dynamically set head sampling on a per-service and operation combination. This is achieved by hosting a jaeger remote sampling endpoint on a collector which with Go custom plugins being able to host OpenTelemetry Go extensions becomes trivial to have FluentBit expose the same functionality (plus reduces re-implementation and helps leverage existing mature community extensions where others can contribute to easily).

As per OpenTelemetry collector documentation:

Extensions provide capabilities on top of the primary functionality of the collector. Generally, extensions are used for implementing components that can be added to the Collector, but which do not require direct access to telemetry data and are not part of the pipelines (like receivers, processors or exporters). Example extensions are: Health Check extension that responds to health check requests or PProf extension that allows fetching Collector's performance profile.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [ ] Example configuration file for the change
• [ ] Debug log output from testing the change

• [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [ ] Run local packaging test showing all targets (including any new ones) build.
• [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [ ] Documentation required for this feature

Backporting

• [ ] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

if the interface is only for OpenTelemetry Collector extensions I would suggest to rename the custom plugin