cloudwatchlogsbeat
cloudwatchlogsbeat copied to clipboard
Add EventId in order to deduplicate
AWS filter logs api call has eventId per message:
events": [
{
"ingestionTime": 1396035394997,
"timestamp": 1396035378988,
"message": "ERROR Event 1",
"logStreamName": "my-log-stream-1",
"eventId": "31132629274945519779805322857203735586714454643391594505"
},
{
"ingestionTime": 1396035394997,
"timestamp": 1396035378988,
"message": "ERROR Event 2",
"logStreamName": "my-log-stream-2",
"eventId": "31132629274945519779805322857203735586814454643391594505"
},
{
While I appreciate the s3 state storage I would rather just reprocess logs for the last X hours and deduplicate by sending a @metadata._id so that elasticsearch output will overwrite the event.
It would be great if the EventId is added as a field so we can further use this information in an ingest pipeline to set the document id. Unfortunately it seems beats can't set elasticsearch document id yet.