amazon-kinesis-agent
amazon-kinesis-agent copied to clipboard
awslabs
Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Hi, I have an ec2 instance running Ubuntu 14.04 with Kinesis agent version 1.1.4 This is my current agent.json config: "flows": [ { "filePattern": "/var/log/nginx/access.log", "deliveryStream": "testing-nginx-kinesis", "initialPosition": "START_OF_FILE", "dataProcessingOptions":...
Hi Experts I use kinesis agent to send Apache Log to a kinesis data stream with simple configuration. { "kinesis.endpoint": "kinesis.us-east-1.amazonaws.com", ... "flows": [ { "filePattern": "/var/log/httpd/access_log*", "kinesisStream": "kinesis-stream-test", "partitionKeyOption":...
does the kinesis-agent have options to gracefully shutdown in case of a kill signal or a `sudo service aws-kinesis-agent stop` signal ? it looks like there is a `shutdownTimeoutMillis` to...
## Description I ran: `git clone --depth=1 https://github.com/awslabs/amazon-kinesis-agent.git && cd amazon-kinesis-agent && sudo ./setup --install` And, error message says: `/home/ubuntu/amazon-kinesis-agent/build.xml:29: Unsupported Java version: 10. Make sure that the Java version...
We see that there is an inconsistent behaviour with the Kinesis Agent. While tailing an application log file we see that behaviour of START_OF_FILE and END_OF_FILE is different. We see...
Hi, I was wondering if there is any way to know if the kinesis agent has completed reading a file. I want to delete the file once after the kinesis...
hi, I am trying to transform a space-separated log into JSON using LOGTOJSON and a match pattern. I don't see any error msg in the kinesis agent logs but the...
When reading a large file from START_OF_FILE, i get out of heap space messages. I have set my max Heap size to 4096m, it worked for smaller files but with...
// sample log -> `[airflowwebserver] [qa] [2019-09-08T19:40:06.087+00:00] {jobs.py:1559} INFO - Harvesting DAG parsing results` agent.json ``` { "cloudwatch.emitMetrics": true, "cloudwatch.endpoint": "monitoring.eu-west-1.amazonaws.com", "firehose.endpoint": "firehose.eu-west-1.amazonaws.com", "kinesis.endpoint": "https://kinesis.eu-west-1.amazonaws.com", "flows": [ { "filePattern": "/var/log/containers/*.log",...
************************************************* "filePattern": "/var/log/wlc.log", "kinesisStream": "ec2_log", "maxBufferAgeMillis": "1000", "dataProcessingOptions": [ { "initialPostion": "START_OF_FILE", "optionName": "LOGTOJSON", "logFormat": "SYSLOG", "matchPattern": "^(\\d+) ([\\w{3} (\\w{3}) (\\d{2}) ([\\d.]+[\\d{2}]) (\\d{4})]) (.*?)", "customFieldNames": ["sq_num", "time_stamp","message" ] ******************************************* my...
