[help-wanted] How to combine csv output plugin with s3?

[javifr]

Hi! Thanks for the great work with this plugin.

I need to output some logs to s3, but in CSV format. ¿Is there a way to combine output plugins?

The concept would be something like this:

{
   "csv" {
     fields => ["name","surname","age"]
     s3 {
     ...
     }
   }
}

If not, the solution would be to add csv format parameter to s3 plugin?

I'm also working on a output plugin for dropbox, completely based in logstash-output-s3, because I need to upload some logs to dropbox in csv format too and i don't know what is the correct approach for the logstash architecture.

Regards!

[nnnnathann]

I'm looking to do this as well. Thoughts?

[javifr]

HI @nnnnathann, I ended up building another plugin based on this one: https://github.com/javifr/logstash-output-s3csv I'm using it on a production environment and everything is working fine.

[nnnnathann]

Perfect, thanks a lot!

[ph]

if the header is not mandatory for you, you can achieve it with 2 differents option:

1. use the plain codec and define the format to match the csv.
2. use a ruby filter to create a new fields on the event and only serialize this field with the plain codec.

The main problem here is the generation of the header, we dont have a generate mechanism to hook into our plugins for that.

[Vinay7820]

Hi javifr, can you please let me know the usage of the plugin which you have developed?

Can your plugin be installed using command line... Something like logstash-plugin install logstash-output-s3csv ??

Steps to intergrate your plugin.

Thank you in advance.

[BradZzz]

@Vinay7820 Instructions on how to run the plugin are in the ruby config. Posting here as it was relatively hard to find. https://github.com/javifr/logstash-output-s3csv/blob/master/lib/logstash/outputs/s3csv.rb

[wssaidong]

How to combine csv output plugin with s3 ？ logstash 8.11.0 version ~~