fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Multiple nestings under the same key lead to duplicate maps

Open Breee opened this issue 9 months ago • 0 comments

Repost of https://github.com/fluent/fluent-bit/issues/3635 that was closed because no one took care of it.

Bug Report

Describe the bug

When you apply multiple Nest filters that target the same key ("put X below foo and then also put Y below foo"), the resulting output drops all those keys except for the one that comes last in the configuration.

I think this is a variation of https://github.com/fluent/fluent-bit/issues/1177 (which was fixed only at the symptomatic level AFAICT, by dropping duplicate keys from json output, but did not change the root cause at all), since according to https://github.com/fluent/fluent-bit/issues/1177#issuecomment-472270159 each nest filter creates a new target map for itself, instead of reusing one that already exists in the record.

Minimal example:

[SERVICE]
    Daemon Off

[INPUT]
    Name dummy
    Tag dummy
    Dummy {"one": "v1", "two": "v2"}
    Samples 1

[FILTER]
    Name nest
    Match dummy
    Operation nest
    Wildcard one
    Nest_under nested
[FILTER]
    Name nest
    Match dummy
    Operation nest
    Wildcard two
    Nest_under nested

[OUTPUT]
    Name http  # You could also use stdout, but the result is a bit less clear for understanding
    Match *
    host localhost  # Run an HTTP echo server like https://gist.github.com/1kastner/e083f9e813c0464e6a2ec8910553e632
    port 8080
    format json
    json_date_key timestamp
    json_date_format double

Expected

[{"timestamp":1623671371.013562,"nested":{"one": "v1","two":"v2"}}]

Actual

[{"timestamp":1623671371.013562,"nested":{"two":"v2"}}]

# with stdout instead of http output, you get this, which is the same as far as JSON semantics are concerned: it does NOT contain the "one" key.
[1623671940.603423664, {"nested"=>{"one"=>"v1"}, "nested"=>{"two"=>"v2"}}]

Seems like you just append to the output instead of using a real map of maps

Breee avatar May 23 '24 15:05 Breee