need some clarification on what this plugin do exactly

[skYl1r]

so from what i understood from the Overview this plugin re-emits events that matches a pattern and let the unmatched events keep throught the rest of the configuration, but when i try something like this :

<source>
 @type tail
  path ./simple.log
  tag log.k8s
  <parse>
    @type none
  </parse>
</source>

<match log.k8s>
   @type rewrite_tag_filter
        <rule>
            key message
            pattern /kalp/
            tag ${tag}.grokfailure
        </rule>
</match>

<match **>
@type stdout
</match>

the events with a kalp string in it get routed with the new tag, pass the rewrite_tag_filter and get catched by stdout plugin, And the events that do not contain kalp will be re-emitted along with the matched events having the same tag and get stuck at rewrite_tag_filter.

so is this how it should work (and if yes, then does it drop the event or re-emit it ?) or is this a bug ? (Thank you)

[okkez]

You can confirm the behavior like the following:

# fluent.conf
<source>
  @type dummy
  tag log.k8s
  dummy [
    {"message": "kalp"},
    {"message": "this is test"}
  ]
  @label @INPUT
</source>

<label @INPUT>
  <match log.k8s>
     @type rewrite_tag_filter
     <rule>
       key message
       pattern /kalp/
       tag ${tag}.grokfailure
     </rule>
     @label @OUTPUT
  </match>
</label>

<label @OUTPUT>
  <match **>
    @type stdout
  </match>
</label>

Run command fluentd -c fluent.conf -vv, and you can see the logs below.

2020-04-05 16:45:39.077616790 +0900 log.k8s.grokfailure: {"message":"kalp"}
2020-04-05 16:45:40 +0900 [trace]: #0 fluent/log.rb:281:trace: rewrite_tag_filter: tag has not been rewritten message="this is test"

The event that is not matched any rules will be consumed by this plugin. If you can catch unmatched events you can add <rule> section like the following:

<rule>
  key message
  pattern /kalp/
  tag ${tag}.grokfailure
</rule>
<rule>
<rule>
  key message
  pattern /.+/
  tag unmatched.${tag}
</rule>

Please read example carefully.

[skYl1r]

You can confirm the behavior like the following:

# fluent.conf
<source>
  @type dummy
  tag log.k8s
  dummy [
    {"message": "kalp"},
    {"message": "this is test"}
  ]
  @label @INPUT
</source>

<label @INPUT>
  <match log.k8s>
     @type rewrite_tag_filter
     <rule>
       key message
       pattern /kalp/
       tag ${tag}.grokfailure
     </rule>
     @label @OUTPUT
  </match>
</label>

<label @OUTPUT>
  <match **>
    @type stdout
  </match>
</label>

Run command fluentd -c fluent.conf -vv, and you can see the logs below.

2020-04-05 16:45:39.077616790 +0900 log.k8s.grokfailure: {"message":"kalp"}
2020-04-05 16:45:40 +0900 [trace]: #0 fluent/log.rb:281:trace: rewrite_tag_filter: tag has not been rewritten message="this is test"

The event that is not matched any rules will be consumed by this plugin. If you can catch unmatched events you can add <rule> section like the following:

<rule>
  key message
  pattern /kalp/
  tag ${tag}.grokfailure
</rule>
<rule>
<rule>
  key message
  pattern /.+/
  tag unmatched.${tag}
</rule>

Please read example carefully.

so does that mean that the event gets stuck circulating in the pipeline and i should implement something like a garbage collector like this :

<match log.k8s>
     @type rewrite_tag_filter
<rule>
  key message
  pattern /kalp/
  tag ${tag}.grokfailure
</rule>
<rule>
  key message
  pattern /.+/
  tag unmatched.${tag}
</rule>
</match>
<match unmatched.**>
@type null
</match>

will this do the trick of getting rid of those events ?

[skYl1r]

@okkez im still waiting for a reply please ?

[y-ken]

@SmittyCooger if there are no pattern to match, it will dropped. so you don't need to create fallback pattern implementation

[mlasevich]

Would be really nice to have a rule (or just rule without a key that will automatically match everything. Using a wildcard pattern (/.+/) is error prone as a - that pattern will not match empty string (easily solved, but still) but more importantly, it breaks if the key is missing.

[Diggsey]

It's insane that this behaviour (dropping unmatched records) is not documented... It's quite reasonable based on the behaviour of other "rewrite" plugins to expect records that don't match a rule to be passed through unchanged.

[jiping-s]

Also a log can be matched by multiple rules which results in multiple versions of it getting emitted. We were using config like:

<rule>
  ...
  tag drop
</rule>
...
<rule>
  ...
  tag next-phase
</rule>

which didn't work as intended because subsequent rules are still evaluated against after the drop rules and logs just got processed as next-phase in the end.

While this behavior is implied by documentation, it's not really clear enough to people not understanding how fluentd works internally.

[LukasJerabek]

@mlasevich Just came by accidentaly looking for something else, but I belive you can somehow utilize the "invert" option and some impossible pattern to rewrite the tag of all to this point unmatched events. Which is not nice, but I think it would work.

[mlasevich]

@mlasevich Just came by accidentaly looking for something else, but I belive you can somehow utilize the "invert" option and some impossible pattern to rewrite the tag of all to this point unmatched events. Which is not nice, but I think it would work.

I am not sure any match, invert or not, would match if key is missing. I solved this problem by using a key i guarantee exists. Ugly, but you do what you have to

[fun2sh]

It's wildly annoying that the documentation doesn't clarify that the plugin drops unmatched event into abyss. I was stuck and frustrated for two days while writing a complex Fluentd configuration and not getting the desired output. Finally, I figured out that for unmatched events, they are simply dropped.

This behavior should be different. The default should be to pass through the unmatched event instead of dropping it. We could have a parameter at each rule level or at the plugin level to indicate whether the unmatched event should be dropped, with the default value set to false.

[y-ken]

Please use pattern /.*/ for final fallback rule

[mlasevich]

@y-ken I think the pattern is not the issue, the issue is that I believe it will not even try that pattern if the key is not there at all

[daipom]

I commented below on what I think needs clarification in the specifications.

• https://github.com/fluent/fluent-plugin-rewrite-tag-filter/pull/16#issuecomment-1980608455