OOTB support AWS Eventbridge

[mjmbischoff]

trafficstars

Proposed commit message

Adding OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] ~I have made corresponding change to the default configuration files~
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

The logic is triggered as a fall-through and is best effort.

Author's Checklist

• [ ] Check for idiomatic golang constructs not being used, with golang not being my primary lang there's a good chance for non-idiomatic constructs to sneak in.

How to test this PR locally

Since it relies on an AWS setup, testing locally beyond units tests is not possible.

Related issues

As it was discovered as part of support ticket there should be an ER linked.

Use cases

AWS has added a new way to notify on S3 changes, next to the existing plain SQS and SNS->SQS, AWS now also support using AWS EventBridge -> SQS

This PR achieves two goals:

• no longer need a custom script (which is the current workaround
• Improve user experience by allowing it to 'work out of the box'

[cla-checker-service[bot]]

💚 CLA has been signed

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @mjmbischoff? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mjmbischoff]

As a general comment; it looks like aws security-lake uses the same format for notifications but omits the detail-type field: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html#sample-notification

Also really liking the part where non of these protocols look like their are versioned or anything. :disappointed:

[mjmbischoff]

As a general comment; it looks like aws security-lake uses the same format for notifications but omits the detail-type field: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html#sample-notification

Also really liking the part where non of these protocols look like their are versioned or anything. 😞

We have an integration for Amazon Security Lake which currently reads:

The Amazon Security Lake integration currently supports only one mode of log collection:

AWS S3 polling mode: Amazon Security Lake writes data to S3, and Elastic Agent polls the S3 bucket by listing its contents and reading new files.

This change would also allow us to improve that integration, again the detail-type field is in question. I guess we should spin up a AWS security lake setup to verify. I hope it's there and the documentation is off as don't think it's safe to assume every event is object created when the field is not available.

The integration is now quite limited as a lot of users have a retention long enough that there's enough objects in the bucket that polling becomes prohibitively expensive.

[mjmbischoff]

https://github.com/elastic/beats/pull/40006/commits/02cae8e1f77d7c3785b0cda6fe3b6a6c7528fa4c to cover 'I have made corresponding changes to the documentation'

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ootb-aws-eventbridge upstream/ootb-aws-eventbridge
git merge upstream/main
git push upstream ootb-aws-eventbridge

[mjmbischoff]

and https://github.com/elastic/beats/pull/40006/commits/a8eb0747be4d37f363e3336cba4c63efe842e6f9 to cover changelog item.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ootb-aws-eventbridge upstream/ootb-aws-eventbridge
git merge upstream/main
git push upstream ootb-aws-eventbridge

[mjmbischoff]

Fixing up / extending tests

• organized imports
• end to end integration test
• removed message array wrapping as it's not the ootb setup, expand_event_list_from_field: Messages unsure what setup was used to get the originally reported format regardless 'expand_event_list_from_field: Messages' would fix this

Think we're good to go.

[mjmbischoff]

@belimawr Can you do a final check?

[pierrehilbert]

Hey @mjmbischoff @belimawr is currently in PTO. Do we have any urgency or should we just wait for him to come back (in one more week)?

[mjmbischoff]

@pierrehilbert As long as it 'stays on the radar' it should be fine.

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ootb-aws-eventbridge upstream/ootb-aws-eventbridge
git merge upstream/main
git push upstream ootb-aws-eventbridge

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ootb-aws-eventbridge upstream/ootb-aws-eventbridge
git merge upstream/main
git push upstream ootb-aws-eventbridge

[pierrehilbert]

@graphaelli (as Baptiste is off) @narph @lalit-satapathy could we have someone in your teams to review this PR please? @alexsapran if you can review for the Ingest Eng Prod part it would be awesome.

[alexsapran]

@alexsapran if you can review for the Ingest Eng Prod part it would be awesome.

I think what prevents our BK from running this PR is the following issue https://github.com/elastic/beats/issues/40503 So, I would rely on manual testing, which I admit I am unfamiliar with this type of testing, so I am not sure my review would be of much value, as it requires only manual testing

[ycombinator]

@mjmbischoff There are some merge conflicts on this PR now. Would you mind rebasing it on main and fixing those conflicts? Thanks!

[mergify[bot]]

backport-8.x has been added to help with the transition to the new branch 8.x. If you don't need it please use backport-skip label and remove the backport-8.x label.