Modify filter not working with Record Accessors

[kharenis]

Bug Report

When using the modify filter to copy nested fields using Record Accessors, the filter fails to find the fields.

To Reproduce

• Configuration:

[FILTER]
    Name                modify
    Match               kube.*
    Copy                $kubernetes['labels']['app']    application
    Copy                $kubernetes['labels']['app.kubernetes.io/name'] application
    Copy                $kubernetes['container_name']   application
    Remove              $kubernetes['pod_id']
    Remove              $kubernetes['docker_id']
    Remove              $kubernetes['container_hash']

• Fluent Bit debug log

[2022/01/21 14:54:40] [debug] [filter:modify:modify.4] Rule COPY $kubernetes['labels']['app'] TO application : No keys matching $kubernetes['labels']['app'] found, not applying rule
[2022/01/21 14:54:40] [debug] [filter:modify:modify.4] Rule COPY $kubernetes['labels']['app.kubernetes.io/name'] TO application : No keys matching $kubernetes['labels']['app.kubernetes.io/name'] found, not applying rule
[2022/01/21 14:54:40] [debug] [filter:modify:modify.4] Rule COPY $kubernetes['container_name'] TO application : No keys matching $kubernetes['container_name'] found, not applying rule

• Output (sent to aws Cloudwatch)

{
    "log": "[14:55:24 INF]  REDACTED",
    "stream": "stdout",
    "time": "2022-01-21T14:55:24.532788648Z",
    "kubernetes": {
        "pod_name": "REDACTED",
        "namespace_name": "REDACTED",
        "pod_id": "723efe40-122a-4b93-960c-f79b09dc583f",
        "labels": {
            "app": "REDACTED",
            "controller-uid": "618de15b-8c23-47a2-8ac8-34cc1d59122d",
            "job-name": "REDACTED",
            "release": "REDACTED"
        },
        "annotations": {
            "kubernetes.io/psp": "eks.privileged"
        },
        "host": "REDACTED",
        "container_name": "REDACTED",
        "docker_id": "59a6665e0e27623e475a49a382e070c63c72b8673e6debcdc63444a455ecb6a9",
        "container_hash": "REDACTED",
        "container_image": "REDACTED"
    }
}

• Steps to reproduce the problem:

Expected behavior 'application' field should be added to the log structure with the value taken from one of the 3 found fields. $kubernetes['pod_id'], $kubernetes['docker_id'], $kubernetes['container_hash'] should be removed from the log structure.

Your Environment

• Version used: Latest
• Configuration:

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[FILTER]
    Name                expect
    Match               kube.*
    key_exists          kubernetes
    key_exists          log
    action              exit

[FILTER]
    Name                grep
    Match               kube.*
    Exclude             $kubernetes['namespace_name']   kube-system

[FILTER]
    Name                stdout
    Match               kube.*

[FILTER]
    Name                modify
    Match               kube.*
    Copy                $kubernetes['labels']['app']    application
    Copy                $kubernetes['labels']['app.kubernetes.io/name'] application
    Copy                $kubernetes['container_name']   application
    Remove              $kubernetes['pod_id']
    Remove              $kubernetes['docker_id']
    Remove              $kubernetes['container_hash']

• Environment name and version (e.g. Kubernetes? What version?): Kubernetes, Latest
• Operating System and version: AWS Linux
• Filters and plugins: kubernetes, expect, grep, stdout, modify, cloudwatch_logs

[nokute78]

https://github.com/fluent/fluent-bit/issues/2152 Currently, filter_modify doesn't support record accessor.

Only STRING:KEY of Conditions supports record accessor. https://docs.fluentbit.io/manual/pipeline/filters/modify#conditions You can set Record Accessor as STRING:KEY for nested key.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[andrewnazarov]

This would be a great help if one wants to work with nested keys. We are facing the same issue trying to modify the k8s metainformation.

[dgsardina]

Same use case and issue, ended up forwarding to fluentd instead.

[chrw]

I'm having the same issue trying to copy the Kubernetes namespace into a variable.

[ryan65]

Hi Anyone found a workaround. I need to remove the pod_id from the kubernetes metadata object, couldnt find a way either with modify or record_modifier No nested support.

[pranavmarla]

Hi Anyone found a workaround. I need to remove the pod_id from the kubernetes metadata object, couldnt find a way either with modify or record_modifier No nested support.

@ryan65 At this point, I suspect a Lua filter is your only option

[ryan65]

Thanks , will try.

[RalfWenzel]

hi, i have the same issue, pod_id and hash values should be removed . Is there any plan to add this as a feature?

[agup006]

@nokute78 for the PR does that help RA with modify?

[sabdalla80]

Just keeping this thread alive, I can also use this feature.

[maxmoshe]

If anybody needs an example for a solution using lua, this worked for me:

[FILTER]
    Name lua
    Match *
    script /fluent-bit/scripts/alter_fields.lua
    call alter_fields

function alter_fields(tag, timestamp, record)
          new_record = record
          new_record["applicationName"] = record["kubernetes"]["namespace_name"]
          new_record["subsystemName"] = record["kubernetes"]["container_name"]
          new_record["kubernetes"]["annotations"] = nil
          return 1, timestamp, new_record
end

Docs: https://docs.fluentbit.io/manual/pipeline/filters/lua

[blackluck]

Anyone have a solution to this kind of problem, without lua? We have lot of problems with lua and want to get rid of it, so that's not an option.

[homiakos]

I support this.

the same grep plugin can work with k8s metadata: https://docs.fluentbit.io/manual/pipeline/filters/grep

In general, it’s strange to change the metadadata format and not support plugins working with the new format

[adrianmace]

This behaviour is necessary. There are so many custom lua scripts floating around doing the same thing.

[brudman]

I agree it would be great if this was supported! Another option for doing this is to use the nest filter described by liyuntao in https://github.com/fluent/fluent-bit/issues/422

[FILTER] Name nest Match kube.* Operation lift Nested_under kubernetes Add_prefix kubernetes.

[FILTER] Name record_modifier Match kube.* Remove_key kubernetes.container* Remove_key kubernetes.pod_id Remove_key kubernetes.docker_id

Then if you need the kubernetes object again:

[FILTER] Name nest Match kube.* Operation nest Wildcard kubernetes.* Nest_under kubernetes Remove_prefix kubernetes.

[LHozzan]

Hi guys.

Same problem here. I have log records (field caller here is array):

{"ts":"2024-10-16T08:55:20.318842569Z","level":"info","caller":{"function":"main.run","file":"/home/runner/work/prometheus-operator/prometheus-operator/cmd/operator/main.go","line":270},"msg":"connection established","kubernetes_version":"1.29.4"}
{"ts":"2024-10-16T08:55:20.514338093Z","level":"info","caller":{"function":"main.run","file":"/home/runner/work/prometheus-operator/prometheus-operator/cmd/operator/main.go","line":351},"msg":"Kubernetes API capabilities","endpointslices":true}

with clashing with another log records type from same component (field caller here is text):

{"caller":"server.go:160","client_ca_file":"/etc/tls/private/tls-ca.crt","err":"stat /etc/tls/private/tls-ca.crt: no such file or directory","level":"warn","msg":"server TLS client verification disabled","ts":"2024-10-16T08:55:21.200935703Z"}
{"caller":"operator.go:302","component":"thanos-controller","level":"info","msg":"successfully synced all caches","ts":"2024-10-16T08:55:21.402699944Z"}
{"caller":"operator.go:590","component":"alertmanager-controller","key":"monitoring/prometheus-stack-kube-prom-alertmanager","level":"info","msg":"sync alertmanager","ts":"2024-10-16T08:55:21.406565648Z"}

Unfortunately, condition in my modify filter not working:

[FILTER]
    Name         modify
    Match        kube.*
    Condition    Key_exists $caller['file']
    Rename       caller caller-attribute

Any news or status? I would like to evade using the Lua scripts, because there is a chance, that over time, the component log records can be changed, so, we must following the changes as well.

[leizhang98]

highly suggest that supported by fluent-bit

[g1franc]

Maybe one day, it will come to ease the life

[imsoumik03]

Please work on this. It would be a great help.