amazon-kinesis-agent
amazon-kinesis-agent copied to clipboard
awslabs
Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Our current flow involves gzipping data client-side and then sending the resulting compressed data to kinesis (to minimize network traffic and prevent shards from being overloaded). Would be great to...
*Issue #, if available:* *Description of changes:* added feature to convert time from one format to another. example of settings in agent.json ``` "dataProcessingOptions": [ { "optionName": "CSVTOJSON", "timeConverter": {...
This is a fix for issue #199 https://github.com/awslabs/amazon-kinesis-agent/issues/199 This code fix replaces the custom CSV parsing done via string manipulation with the very flexible Apache Commons CSV project. An additional...
*Issue #, if available:* #214 *Description of changes:* now in agent.json in `dataProcessingOptions` section we can add ignored fields ``` "dataProcessingOptions": [ { "optionName": "CSVTOJSON", "ignoredFieldNames": [ "column1", "column2", "column3"...
Business problem. In my case we have source csv files with many fields. And these files are read by many apps, each apps require different columns. In kinesis we do...
*Issue #172* *Description of changes:* Add support of [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns in directories. Log files: - `/dir/2020-10-14/log1.txt` - `/dir/2020-10-12/log2.txt` can be parsed with `filePattern` - `/dir/*/log*.txt` By submitting this pull request,...
*Issue #, if available:* *Description of changes:* Added sample dockerfile to help with containerization of the agent. By submitting this pull request, I confirm that you can use, modify, copy,...
currently using a kinesis endpoint of the form `https://kinesis-ae1.foo.bar.baz.org` and when I start the agent, it reads credentials correctly from env vars (ie `AWS_SECRET_ACCESS_KEY`, `AWS_ACCESS_KEY_ID`), however it will not use...
Hi, Unlike the KPL there does not seem to be any userRecord aggregation. This means that we very quickly get throttled on events. Am I missing something? Without this the...
We are using kinesis agent to push events to firehose by reading a log file. This setup has been running file for couple of years. On May 27th we noticed...
