fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

in gelf: new plugin

Open Matanelc opened this issue 2 years ago • 4 comments

Rebased on -> https://github.com/fluent/fluent-bit/pull/4156

New input plugin for ingest GELF messages from TCP and UDP. In UDP support for gzip and zlib compressed messages, and chunked messages.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Example configuration file for the change

  • [N/A] Debug log output from testing the change

  • [x] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • [x] Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Matanelc avatar Apr 14 '23 14:04 Matanelc

Can you link the docs PR?

patrick-stephens avatar May 05 '23 16:05 patrick-stephens

Can we add any integration tests? https://github.com/fluent/fluent-bit-ci

patrick-stephens avatar May 05 '23 16:05 patrick-stephens

I'm assigning this to myself so I can properly review it on monday.

leonardo-albertovich avatar May 05 '23 17:05 leonardo-albertovich

It seems to me that this PR is packed with goodies but I think it needs to be split in three :

  1. The changes in flb_gzip.c : for this one in particular I think the description should explain the changes it makes to flb_gzip_uncompress because @cosmo0920 is in the process of making some changes to it and I don't want this to cause unnecessary / avoidable hardships.
  2. The gelf unpacker code should be submitted as a stand alone "component" just like flb_pack_gelf and ideally if possible a parser that uses this new unpacker should be added. That way this new feature can be easily reused .
  3. If by this point we determine that in_tcp and in_udp can't be improved to use the newly created gelf parser (or if we determine that the addition would improve ease of use greatly) a third PR with the output plugin should follow.

Obviously this is the "best case scenario" and I understand that not everyone might share these opinions but I do think we should at least try to find a middle ground to make the most out of this.

leonardo-albertovich avatar May 08 '23 15:05 leonardo-albertovich