S3 slow down and connection refused problems

[txau]

I believe the intermittent errors we see at S3 with S3 timeout are related to a limitation in the rate.

If there is a fast spike in the request rate for objects in a prefix, Amazon S3 might return 503 Slow Down errors while it scales in the background to handle the increased request rate. To avoid these errors, you can configure your application to gradually increase the request rate and retry failed requests using an exponential backoff algorithm [1].

From our latest logs, we see some slowdown errors, in the form of:

7:58 PM: 2024-10-07T19:58:33.712Z [tenant] requestId: 6529 
url: /api/files/1714673803529gdetwwu5oe9.jpg
SlowDown: UnknownError
    at throwDefaultError (/opt/uwazi/cores/core-1.187.3/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /opt/uwazi/cores/core-1.187.3/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/opt/uwazi/cores/core-1.187.3/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4743:14)

Then:

url: /api/files/17216202011795hfogmyqftu.pdf
Error: connect ECONNREFUSED IP_address:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -111,
  code: 'ECONNREFUSED',

Either we catch the errors and handle the exception or put in place other mitigation measures to handle the high rating.