Amazon S3 input

[megastef]

trafficstars

Logagent should have a plugin

1. to read S3 data as soon new files are created
2. to import historical data

[prog8]

• For initial dump Logagent probably needs to ls a bucket and read all files
• For new data Logagent probably has to subscribe to SNS

[otisg]

... or simply check periodically?

[prog8]

I think a periodical check is no-go. A bucket can have thousands, tens thousands or hundreds of thousands of objects and ls will not be fast enough.

[megastef]

Perhaps we just need an example configuration and documentation to combine s3cmd with other Logagent plugins:

• Logagent command plugin to start S3-Sync http://s3tools.org/s3cmd-sync s3cmd sync --exclude '*.zip' --include 'dir2/*.log' . s3://s3tools-demo/demo/
• Logagent file input plugin to read downloaded files
• Logagent Elasticsearch output plugin to stream files to Sematext Cloud or Elasticsearch ...

[megastef]

Documentation links for plugins: https://sematext.com/docs/logagent/#plugins

[megastef]

Helpful S3 download module: https://www.npmjs.com/package/s3-download-stream