Support enriching received data with additional global labels

[axw]

trafficstars

In 8.0, when we replaced the libbeat Elasticsearch output, we dropped support for the fields configuration.

The main reason we dropped support for fields was so we could move away from the free-form bag-of-key-value pairs libbeat model, which required inefficient reflection-based encoding. At that time we expected users could add fields with an ingest pipeline.

As it turns out, some users want to run multiple APM Servers, and have data enriched according to which APM Server receives the data. For example, one may have an APM Server per organisational department (IT, engineering, etc.), and label data accordingly.

To enable this, we should either reinstate fields, or introduce a new mechanism for setting global labels like Jaeger agent tags.

[axw]

either reinstate fields

This would not be unreasonable, as we're considering pass-through of arbitrary OpenTelemetry attributes. To do that, we'll flatten all fields (so foo.bar.baz is written as a dotted field name, rather than a hierarchical object), and set ignore_malformed for all fields. That way users won't be able to break ingestions; they may just have some fields missing when they conflict, the likelihood of which happening will be minimised as much as feasible.

Still, I think the configuration being about setting global labels would better match the intention.

[simitt]

Supporting tags sounds reasonable. When thinking about using APM Server managed by Fleet, users would create one Elastic Agent policy with APM and then enroll multiple Elastic Agents into this policy. For supporting per apm server configurable labels, we probably would need to support a predefined set of variables, such as host.name.

[axw]

@simitt do they need to variable? If someone wanted to do processing based on the apm-server instance, they could use observer.hostname in their ingest pipeline.

What I had in mind was more use-case oriented. Like you might have some APM Server instances that are dedicated to Department X. Those APM Servers should add labels.department: x to all documents they index. This could be done on Fleet with multiple policies.