`modify` filter does not respect nested keys

[astorath]

Bug Report

Describe the bug

modify filter does not respect nested keys

To Reproduce

• Config:

  [SERVICE]
      Flush        1
      Daemon       Off
      Log_Level    debug
      Parsers_File parsers.conf
  
  [INPUT]
      Name             tail
      Path        /log.json
  
  [FILTER]
      Name parser
      Match *
      Key_Name log
      Parser test
  
  [FILTER]
      Name  modify
      Match *
  
      Condition Key_exists           json
  
      Copy        json.Message      log
  
  [OUTPUT]
      Name  stdout
  

• Example log message if applicable:

{ "json": { "Message": "HttpClientTracingProvider:: Class is receiving HTTP Response from  using .: StatusCode=200 Elapsed=00:00:00.0078867 ResponseHeaders={$Headers} ResponseBody={$Body}" } }

• Steps to reproduce the problem: Just run fluent-bit with config attached.

Expected behavior

json.Message contents should be copied to log key.

Screenshots

Your Environment

• Version used: 1.3.7, 1.4.3
• Configuration: provided above
• Environment name and version (e.g. Kubernetes? What version?): docker 19.03.8
• Server type and version: -
• Operating System and version: MacOS 10.15
• Filters and plugins: -

Additional context

Non-nested keys are working fine

[mshade]

Can confirm this still does not work on v1.5.2. What I'm doing to work around, for now, is using the nest filter to lift records up and then process with rename. This isn't optimal, though, as I'd ideally only like to lift 1 or 2 keys rather than the whole nested structure.

[vinodkmrm]

Does not work in v1.5.2 too. I had to use the same workaround.

[trnl]

Anyone found a way out? Any relation with https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/record-accessor?

[vladimir259]

The issue still exist in 1.7 :-(

[maximchukm]

I've found out a more effective workaround for this issue than using nest filter, you can use instead the Lua filter https://docs.fluentbit.io/manual/pipeline/filters/lua with a script like this:

function label_key_rename(tag, timestamp, record)
    local old_key, new_key = "oldKey", "newKey"

    local labels = record["kubernetes"]["labels"]
    local new_record = record
    local code = 0

    if labels[old_key] ~= nil then
        code = 1
        local new_labels = {}
        for key, value in pairs(labels) do
            if key == old_key then key = new_key end
            new_labels[key] = value
        end
        new_record["kubernetes"]["labels"] = new_labels
    end

    return code, timestamp, new_record
end

[astorath]

I've found out a more effective workaround for this issue than using nest filter, you can use instead the Lua filter https://docs.fluentbit.io/manual/pipeline/filters/lua with a script like this:

I'm a bit concerned about using lua - the main point of using fluent-bit in place of fluentd is performance. Falling back to scripting somewhat defeats this advantage. Am I wrong?

[maximchukm]

I've found out a more effective workaround for this issue than using nest filter, you can use instead the Lua filter https://docs.fluentbit.io/manual/pipeline/filters/lua with a script like this:

I'm a bit concerned about using lua - the main point of using fluent-bit in place of fluentd is performance. Falling back to scripting somewhat defeats this advantage. Am I wrong?

Sure, native C implementation still will be faster. But using lua script with straight logic is much better both from readability and performance points than juggling with "nest" plugin (one operation per field instead of 3+)

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[isaccavalcante]

Still a valid issue, commenting so this doesn't get closed.

[mattwhittingham]

Not sure how you would support this perfectly without a new parameter with a json path. I've used nest like this, hopefully this helps people.

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          temp_l1_

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        temp_l1_labels
        Add_prefix          temp_l2_

    [FILTER]
        Name                modify
        Match               kube.*
        Rename              temp_l2_label1 label1
        Rename              temp_l2_label2 label2

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           nest
        Wildcard            temp_l2_*
        Nest_under          temp_l1_labels
        Remove_prefix       temp_l2_

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           nest
        Wildcard            temp_l1_*
        Nest_under          kubernetes
        Remove_prefix       temp_l1_

[VIPHercules]

function label_key_rename(tag, timestamp, record) local old_key, new_key = "oldKey", "newKey"

local labels = record["kubernetes"]["labels"]
local new_record = record
local code = 0

if labels[old_key] ~= nil then
    code = 1
    local new_labels = {}
    for key, value in pairs(labels) do
        if key == old_key then key = new_key end
        new_labels[key] = value
    end
    new_record["kubernetes"]["labels"] = new_labels
end

return code, timestamp, new_record

end

I'm attempting this approach, however I'm completely unable to find any documentation on how to add my lua file to fluent bit. We are using the docker image, do we have to create a custom image on top of the base fluentbit image and upload the lua file there?

[scila1996]

I have the same issue ! i running fluent-bit from container image 1.8.6 latest

Input

{
  "method":"POST",
  "remoteAddr":"1.2.3.4",
  "user":{"group": ["a", "b", "c"]}
}

    [FILTER]
        Name    record_modifier
        Match   app.log
        Remove_key  remoteAddr
        Remove_key  user.group

Remove the nested key user.group not work

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[amitjoseph]

This issue is still valid, why isn't this issue not labeled as a bug?

[data-dude]

Still a problem. I'd like to remove some kubernetes.examplekey but the filter doesn't respect nested keys. record_modifier has the same problem

[dFurman]

Still a problem. I'd like to remove some kubernetes.examplekey but the filter doesn't respect nested keys. record_modifier has the same problem

I encountered a similar issue and i managed to resolve it using a super simple lua script, hope it helps:

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

      return 2, timestamp, record
    end

and you use it like this:

[FILTER]
        Name    lua
        Match   kube.*
        script  functions.lua
        call    remove_k8s_labels_annotations

[sane4ek-2]

This problem is still actual not only for the modify plugin but GELF output plugin also. I cannot set a nested key for Gelf_Host_Key parameter.

[alt-dima]

But you can simply turn it off

[FILTER]
    Name                kubernetes
    Labels              Off
    Annotations         Off

Still a problem. I'd like to remove some kubernetes.examplekey but the filter doesn't respect nested keys. record_modifier has the same problem

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[TheZhyer]

But you can simply turn it off

[FILTER]
    Name                kubernetes
    Labels              Off
    Annotations         Off

Still a problem. I'd like to remove some kubernetes.examplekey but the filter doesn't respect nested keys. record_modifier has the same problem

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

Turning it Off does not work. The K8S labels and annotations still remain present.

It would be nice if we could turn it Off so it does not show anymore, Or to be able to target nested keys in case we want to remove specific keys only. Commenting as this issue is still present.

[antoniocascais]

Having the same problem as explained in this issue.

My use case: we have a field (that comes from kubernetes audit logs, so we can't modify the source) where record["foo"]["bar"] sometimes is a specific string and other times is a json object. Because of this, Opensearch complains and rejects the log.

Our idea is to modify the record. Something like

if record["foo"]["bar"]=string then 
  modify record to record["foo"]["custom"]["bar"]
end

I know this can be done with a Lua script, but it would be much more readable (and less error prone) to do it via the modify filter.

Any plans on adding this feature to modify?

[mjiderhamn]

Please note that fixing this could have side effects.

For example, with ElasticSearch there will be an error if for some entries foo has nested values and for some it is a string; i.e. object mapping for [foo] tried to parse field [foo] as object, but found a concrete value.

Currently, this can be avoided by renaming keys with a string value like so

      [FILTER]
          Name modify
          Rename foo bar
          Condition Key_exists foo

[data-dude]

Still a problem. I'd like to remove some kubernetes.examplekey but the filter doesn't respect nested keys. record_modifier has the same problem

I encountered a similar issue and i managed to resolve it using a super simple lua script, hope it helps:

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

      return 2, timestamp, record
    end

and you use it like this:

[FILTER]
        Name    lua
        Match   kube.*
        script  functions.lua
        call    remove_k8s_labels_annotations

Yea that might work for some people. I found the easiest was to set the kubernetes plugin filter to annotations off and lables off See docs for more details: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes