Storing API Log to Elasticsearch failing for being too large

[chrismshea]

Issue

We utilize Elasticsearch to store and access API logs. When reviewing logs I found a number of log statements stating that the API log was unable to be stored in Elasticsearch because the content length was too large. If this continues we will lose access to historical API requests which is critical to supporting our product.

Failed to store API log to Elasticsearch. Exception message: Error executing "SendMessage" on "https://sqs.us-east-1.amazonaws.com/624083294473/log-queue-prod-us-east-1"; AWS HTTP error: Client error: `POST https://sqs.us-east-1.amazonaws.com/624083294473/log-queue-prod-us-east-1` resulted in a `413 Request Entity Too Large` response: HTTP content length exceeded 1662976 bytes. Unable to parse error information from response - Error parsing XML: String could not be parsed as XML

@jmymy states:

The minimum message size is 1 byte (1 character). The maximum is 262,144 bytes (256 KiB).
Im guessing we are hitting these limits

Search Link

https://k1-prod-us-east-1.subscribepro.com/_dashboards/app/data-explorer/discover#?_a=(discover:(columns:!(_sourc[…]20log%20to%20Elasticsearch%22'))