OpenTelemetry Plugins should have a way to modify Resource and Scope

[braydonk]

Is your feature request related to a problem? Please describe. Resource Attributes in OTLP are used to identify logs with a particular resource, making them very important for an OTLP exporter to be able to modify. Today, there is no way to add Resource Attributes, but this is very common in OTLP pipelines and is crucial for making OTLP useful for more cases in Fluent Bit.
Scope is not quite as crucial as Resource, but it would still make sense to allow access to that too.

Describe the solution you'd like Something similar to add_label could work. add_resource_attribute perhaps. I think that would still be too limiting though, and the best possible solution is allowing a way to operate on resource in all the standard field modification operations.

With Scope, it would make sense to actually instrument a Fluent Bit scope, so that downstream consumers will be able to tell that the data was produced by a Fluent Bit agent. Other than that, the same solution as Resource should be available for Scope too.

Describe alternatives you've considered

N/A

Additional context

To make Fluent Bit as useful as possible for OTLP pipelines, it really should be able to operate on Resource and Scope just like the OpenTelemetry Collector can. See also #8205, which is actually a bug particular to logs that definitely makes Fluent Bit hard to use for this case.

[liji-canva]

Totally agree with this feature request. Do we any plan on this?

The current workaround is to setup an intermediate opentelemetry collector between fluent-bit and the final opentelemetry collector and configure the resource processor on this intermediate opentelemetry collector to add resource attributes. This obviously makes things complicated.

[SHaaD94]

What about altering syntax of add_label to be following for out:opentelemetry?

add_label            label_name label_value resource|scope ?

Would be happy to raise a PR if this will get approved

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[braydonk]

This is still relevant.

[shaohme]

It is relevant

There is also a use case in which fluent-bit collects metrics from all kinda source on a "device" with a unique ID. Instead of all programs separately declares the same resource attribute, like "service.instance.id", Fluent-Bit could add this attribute value to all Spans/Log/Metrics it collects before it outputs them to a remote OpenTelemetry Collector.

[edsiper]

FYI: Opentelemetry Logs resource and scope handling is done through #8989

[edsiper]

https://github.com/fluent/fluent-bit/pull/8989 merged

[shaohme]

#8989 merged

Nice to see it implemented for logs :+1: but shouldn't we close the issue when support for metrics and spans have been implemented too?

[Brodiemm]

Yes, appreciate the work on logs, is there an open ticket for metrics and traces resource attribute modification support?