logstash-input-cloudwatch-logs icon indicating copy to clipboard operation
logstash-input-cloudwatch-logs copied to clipboard

Use dynamodb for sincedb storage

Open carolinebeauchamp opened this issue 6 years ago • 2 comments

So, running in a container, I can't rely on a file system based sincedb - so I've written this to store sincedb in dynamodb instead. It needs suitable AWS permissions to create a table, read items and put items.

New configuration parameter:

dynamodb_sincedb_table : name of dynamodb table to use - it will be created if it doesn't already exist.

And if that parameter exists then it will ignore all the sincedb file, and use dynamodb

TODO: Need to work out how to do rspec tests on this, and the code probably needs some tidying

Thanks!

carolinebeauchamp avatar Jul 03 '18 12:07 carolinebeauchamp

This is awesome! I was just thinking of implementing this and now I don't have to. Looking forward to getting this merged in.

ajmath avatar Jul 17 '18 14:07 ajmath

Thanks for the contribution!

It looks like the failing tests are due to some upstream changes to the logstash-mixin-aws package, so I think we're safe there - though I haven't had the time to really build out the test cases as I would like on this project.

I do have a couple concerns, but overall it looks like a good start.

lukewaite avatar Jul 18 '18 23:07 lukewaite