elastic
[OTel] Data Serialization fails on `struct` types
Describe the enhancement:
For normal Beats metricsets, you can add a custom Go struct objects to the outgoing mapstr.M as part of the Beat Event object. As long as the objects support serialization, Beats will do the right thing and push the data
However, for OTel, via otelconsumer, these objects fail to be serialized and you end up with <Invalid value type %T> being logged per event (where %T becomes the root data type, which is unhelpful).
https://github.com/elastic/beats/blob/a80b60a811aa1243bce4fb73885a941d7d546893/x-pack/libbeat/outputs/otelconsumer/otelconsumer.go#L148-L150
A relatively simple example exists in the autoops_es/cat_shards metricset:
https://github.com/elastic/beats/blob/a80b60a811aa1243bce4fb73885a941d7d546893/x-pack/metricbeat/module/autoops_es/cat_shards/data.go#L104-L118
This pushes a mapstr.M that contains a slice of a custom struct object and, within each struct, is another array (up to 4) of structs representing shards. The current OTel handling fails to process this event and instead sends an array of "node_index_shards":[null,null,null,null,null], which is obviously unhelpful.
Describe a specific use case for the enhancement or feature:
Existing metricsets should work as-is for OTel conversion and nested structs that are serializable should be automatically converted to map[string]any (or []map[string]any -> []any) so that OTel (via pdata) does not have any issues.
Relatedly, OTel should also be capable of handling other map types like map[string]int, but it can only handle map[string]any (aka map[string]interface{}). I'm less concerned with map[T]any type variants, but if it's a map with a string key, we should probably support it.
Workaround:
For developers hoping to use their metricset(s) in OTel, there is a relatively simple workaround, but it may require updating tests: marshal your struct to JSON, then unmarshal it back as a map[string]any before sending it to libbeat.
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
Unfortunately, no. I had a separate PR that used the workaround described above (https://github.com/elastic/beats/pull/45015), but I did not implement a core fix for it.
The only solution here that I am aware of would be to add a reflective check to see if the type can be serialized (or blindly attempt to do so) in the default case (you can see what I did for the workaround:
https://github.com/elastic/beats/blob/7bfa78e27177ef1ef8d3e52254c941eb932d9542/x-pack/metricbeat/module/autoops_es/cat_shards/struct_to_map.go#L12-L27
If that produces a map[string]any (in a more generalized conversion), then we'd need to recursively handle that to ensure that any nested arrays became []any.
For normal Beats metricsets, you can add a custom Go struct objects to the outgoing mapstr.M as part of the Beat Event object. As long as the objects support serialization, Beats will do the right thing and push the data
It seems clear to me that otelmap should do what the classic beats does in this case.
It seems clear to me that otelmap should do what the classic beats does in this case.
It looks like the libbeat version of this is much more complex:
https://github.com/elastic/beats/blob/0dd36b9c6031fb50deb0a2cef9cbd85861a87e9a/libbeat/common/event.go#L144-L255
It seems clear to me that otelmap should do what the classic beats does in this case.
It looks like the
libbeatversion of this is much more complex:
Indeed, I was just looking at this exact code. If we want feature parity between Beats and receivers, then we should include this conversion logic to match. From what I can see, a good portion of it is very similar to what we already have in otelmap. I'm still investigating.