fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

Allow compressed data for tcp/udp/syslog inputs

Open srilumpa opened this issue 2 years ago • 0 comments

Is your feature request related to a problem? Please describe.

To reduce network congestion, we want to enabled log compression when forwarding logs to our fluentd server. Fortunately, most of our tools can handle sending compressed syslog or raw data. Unfortunately, fluentd doesn't and takes the data "as-is". It appears in buffer files as strings containing only escaped unicode characters (ex: =\u0001\u0000).

Describe the solution you'd like

It would be great if we could setup tcp, udp or syslog inputs to receive compressed data. Having a compress directive accepting as values a boolean (true/false) or a compression algorithm (none/gzip/...).

<source>
  @type tcp
  port 3333
  bind 0.0.0.0
  tag input.myapp
  compress gzip
  <format>
    @type none
  </format>
</source>

Describe alternatives you've considered

So far we managed to activate compression on our data by logging our tools activities into files and by using a local td-agent tailing each file and forwarding their contents to our td-agent central server with compression enabled.

Additional context

No response

srilumpa avatar May 02 '22 14:05 srilumpa