cloudwatchlogsbeat icon indicating copy to clipboard operation
cloudwatchlogsbeat copied to clipboard

Add EventId in order to deduplicate

Open wernerb opened this issue 4 years ago • 0 comments

AWS filter logs api call has eventId per message:

events": [
    {
      "ingestionTime": 1396035394997,
      "timestamp": 1396035378988,
      "message": "ERROR Event 1",
      "logStreamName": "my-log-stream-1",
      "eventId": "31132629274945519779805322857203735586714454643391594505"
    },
    {
      "ingestionTime": 1396035394997,
      "timestamp": 1396035378988,
      "message": "ERROR Event 2",
      "logStreamName": "my-log-stream-2",
      "eventId": "31132629274945519779805322857203735586814454643391594505"
    },
    {

While I appreciate the s3 state storage I would rather just reprocess logs for the last X hours and deduplicate by sending a @metadata._id so that elasticsearch output will overwrite the event.

It would be great if the EventId is added as a field so we can further use this information in an ingest pipeline to set the document id. Unfortunately it seems beats can't set elasticsearch document id yet.

wernerb avatar Dec 16 '20 21:12 wernerb