notification-api icon indicating copy to clipboard operation
notification-api copied to clipboard

Add validation to check that notifications dont exceed sqs msg limit

Open whabanks opened this issue 1 year ago • 1 comments

Summary | Résumé

Notifications payloads sent to /v2/notifications/{type} are now checked to ensure that their size in bytes does not exceed the SQS message payload size limit.

  • Add logging that identifies the largest piece of data when oversized payloads are attempted.
  • Add a message informing the API user of the size limit and pointing them to the largest item in their payload as a guide.
  • Add a recursive utility method to flatten a dict for easier processing in some situations.

SQS has a payload limit 262,144 bytes, or 256Kib. When Celery attempts to consume inflights that are greater than this size limit it will fail and the message is returned to the inflight inbox. This loop continues until the inflight is expired thus creating a lot of unnecessary Celery errors in our logs.

whabanks avatar Jan 18 '24 21:01 whabanks