logstash-output-s3
logstash-output-s3 copied to clipboard
logstash-plugins
Support for dynamic prefix field based s3 directory separation - incorporating review comments
I have signed the CLA both from my patch id and git hub id. Not sure why it shows the error.
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.
Where does this stand?
I'm on the brink of deploying a new log setup for our company, and just now realized that there is no support for dynamic prefixes - kind of a deal breaker.
Will this be rolled out any time soon?
@nigoel Hmm.. The patch has a commit from your adobe email, but in the CLA db I see a gmail address. I think this is the problem with the CLA check. Can you fix the commit or sign with your adobe email?
I like the idea. Can you tell me a story about the no_event_wait setting? It's not clear to me what the purpose of it is, but it's possible I didn't read deeply enough. Help? :)
@jordansissel It looks like a rather simple way to make sure that the nested directory paths that were created by the 'prefix' are cleaned up on the local filesystem some time after the files have been uploaded to S3.
In this case, it waits for "time_file" to expire "no_event_wait" times on files within the "prefix" directory, and if each consecutive time the files it scans are empty, up to "no_event_wait" times, then it purges the local directories.
It is a bit of an awkward setting to explain and probably even more so to expose to users without more clarity. But essentially, if the files in the local "prefix" directory are continuously empty for a period of time, purge them.
Of course, I could be entirely wrong about it since this code is new to me. But this is a feature I would really like to see added, if possible.
