[main] Fix handling of custom Endpoint when using S3 + SQS

[strawgate]

Fix issues described in https://github.com/elastic/beats/issues/39706 that prevent using a custom endpoint with S3 + SQS.

Users can workaround this issue via S3 bucket polling. The S3 bucket polling still works just fine with a custom endpoint, it's just adding in SQS where it breaks. We need to publish a https://github.com/elastic/integrations/pull/9865

Work required for Main:

• [ ] Only use a custom Endpoint Resolver if the Endpoint doesn't start with S3.
• [ ] Parse Endpoints with custom domains in the format of https://s3.us-east-1.abc.xyz so users don't have to specify a region

[botelastic[bot]]

This pull request doesn't have a Team:<team> label.

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @strawgate? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[strawgate]

@faec can we make sure these items are fixed in your refactor as well? i.e. the ones in https://github.com/elastic/beats/pull/39709 that arent already fixed?

[dinofizz]

We currently use AWS VPC interface endpoints for SQS and S3. For SQS we do have "Private DNS names" enabled, and so we do not need to set an endpoint for SQS (i.e. traffic to sqs.eu-west-1.amazonaws.com is automatically routed to the internal VPC endpoint). However we do not have private DNS names enabled on S3 interface endpoints, and so here our endpoint hostname is https://bucket.vpce-07bedb45c8767nd.s3.eu-west-1.vpce.amazonaws.com.

It's not obvious to me from the comments if this change will enable us to use the input in this way. Ideally we would just set (or omit) an endpoint per AWS service, which would allow us to use a "https://bucket.vpce-..." endpoint for S3 and not use an endpoint for SQS.

[strawgate]

this was not supposed to merge, it was just an example for how to include it post s3/sqs refactor, ill close the PR

[strawgate]

We currently use AWS VPC interface endpoints for SQS and S3. For SQS we do have "Private DNS names" enabled, and so we do not need to set an endpoint for SQS (i.e. traffic to sqs.eu-west-1.amazonaws.com is automatically routed to the internal VPC endpoint). However we do not have private DNS names enabled on S3 interface endpoints, and so here our endpoint hostname is https://bucket.vpce-07bedb45c8767nd.s3.eu-west-1.vpce.amazonaws.com.

It's not obvious to me from the comments if this change will enable us to use the input in this way. Ideally we would just set (or omit) an endpoint per AWS service, which would allow us to use a "https://bucket.vpce-..." endpoint for S3 and not use an endpoint for SQS.

The endpoint handling is all or nothing, so I do not believe this change does not offer setting a specific endpoint for one component or the other.