Unable to decode json in a nested field

[nerzhul]

Bug Report

Describe the bug

Hello,

It's more a support request than a bugfix. i'm trying to port parts of our historical fluentd configuration to fluentbit (great job for the compat on this).

While json decoding is working, i'm trying to keep compat with our existing codebase and our customer dashboards. On fluentd we decode json appending a json. key prefix. I didn't found any option to perform this on fluentbit while decoding json. Is this a possible thing ?

Thanks

[nerzhul]

To be more precise, i'd like to achieve similar thing like on fluentd

<filter foo.bar>
  @type parser
  key_name log
  reserve_data true
  inject_key_prefix data.
  <parse>
    @type json
  </parse>
</filter>

Currently the inject_key_prefix is mssing, then it merge the json with all already present metadata field retrieves from the system itself or decoded from the log filename in containerd, which can overwrite them

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[nerzhul]

it's clearly a needed feature as lua is not sufficient for all cases, especially when we have conflicting metadata from the record and the parsed json

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[RaJiska]

Bump.

This is something I am also interested about as well. Could you find a solution?

Not having this brings up additional challenges depending of the output. For example when using Elasticsearch, the format for an index needs to be consistent where key needs to be of the same format (for example they can't be of string type in a message, and int type in another one).

Since we use Fluentbit to parse multiple containers logs, when using JSON parsing of the log key we may end up with the same key name parsed at the root but with different types, hence the need for some sort of namespacing.

This is without even accounting for key name clash which may arise (e.g, a parsed JSON structure having the same key name as an already existing key at the root).

[zephinzer]

bump, this would be really useful 🙏🏼

[RaJiska]

Opened a PR to implement this feature #9828

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.