Application Pipeline: Check the Maximum size of HTTP request payloads of AOS and alert

[Paul-AWS]

Describe the feature

Different AOS worker instance type has different value of Maximum size of HTTP request payloads, which may cause "Bulk Load failed: {"Message": "Request size exceeded 10485760 bytes"} " error if the size of S3 file is larger than the maximum size of HTTP request payloads. It will be better if there's a check when creating application pipeline, and pop a warning/error when the user is try to create a pipeline with a buffer size that is larger than the AOS maximum size of HTTP request payloads.

The list of maximum size of HTTP request payloads in different instance type: https://docs.aws.amazon.com/en_us/opensearch-service/latest/developerguide/limits.html#network-limits

Use Case

For example, the default buffer size is 50MB when you create an application pipeline using S3 as buffering. If the customer choose m6.large as their AOS worker instance type (Maximum size of HTTP request payloads: 10MB), all the Lambda will failed to write logs into AOS with the error "Bulk Load failed: {"Message": "Request size exceeded 10485760 bytes"} ", which means no logs or lack of logs in AOS and a big cost due to lots of Lambda execution and retry execution.

Proposed Solution

No response

Other Information

No response

Solution version used

1.0.0