fluent-bit
fluent-bit copied to clipboard
Multiple nestings under the same key lead to duplicate maps
Repost of https://github.com/fluent/fluent-bit/issues/3635 that was closed because no one took care of it.
Bug Report
Describe the bug
When you apply multiple Nest filters that target the same key ("put X below foo and then also put Y below foo"), the resulting output drops all those keys except for the one that comes last in the configuration.
I think this is a variation of https://github.com/fluent/fluent-bit/issues/1177 (which was fixed only at the symptomatic level AFAICT, by dropping duplicate keys from json output, but did not change the root cause at all), since according to https://github.com/fluent/fluent-bit/issues/1177#issuecomment-472270159 each nest filter creates a new target map for itself, instead of reusing one that already exists in the record.
Minimal example:
[SERVICE]
Daemon Off
[INPUT]
Name dummy
Tag dummy
Dummy {"one": "v1", "two": "v2"}
Samples 1
[FILTER]
Name nest
Match dummy
Operation nest
Wildcard one
Nest_under nested
[FILTER]
Name nest
Match dummy
Operation nest
Wildcard two
Nest_under nested
[OUTPUT]
Name http # You could also use stdout, but the result is a bit less clear for understanding
Match *
host localhost # Run an HTTP echo server like https://gist.github.com/1kastner/e083f9e813c0464e6a2ec8910553e632
port 8080
format json
json_date_key timestamp
json_date_format double
Expected
[{"timestamp":1623671371.013562,"nested":{"one": "v1","two":"v2"}}]
Actual
[{"timestamp":1623671371.013562,"nested":{"two":"v2"}}]
# with stdout instead of http output, you get this, which is the same as far as JSON semantics are concerned: it does NOT contain the "one" key.
[1623671940.603423664, {"nested"=>{"one"=>"v1"}, "nested"=>{"two"=>"v2"}}]
Seems like you just append to the output instead of using a real map of maps