Cloudfront Invalidation rate limit issues

[dipakjain94]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone through Developer Guide and API reference
• [x] I've checked AWS Forums and StackOverflow for answers
• [x] I've searched for previous similar issues and didn't find any solution
• [x] This is an issue with version 2.x of the SDK

Describe the question We have an application that is using S3 and cloudfront. we have define a process which will create or replace new object/files in S3 bucket which will consume in our mobile app with cloudfront URL.

Process - Once we get a file in S3 bucket, we triggered lambda function that execute invalidation for specific S3 object (without using * wildcard character).

Problem - we are getting Throttling: Rate exceeded exception for invalidation request. we observe that first 6-7 request executed successfully then we receive this error.

According to AWS documentation - we can have invalidation requests for up to 3,000 files per distribution in progress at one time. but we are receiving error after 6-7 request.

**Exception**
*Throttling: Rate exceeded
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'Throttling',
  time: 2021-12-07T08:13:27.058Z,
  requestId: '*****',
  statusCode: 400,
  retryable: true
}*

SDK version number v2.981.0

[vudh1]

Hi @dipakjain94 thanks for reaching out. Can you send a reproducible code that we can use to investigate this more?

[dipakjain94]

Hi @vudh1 Below are the lambda handler code which is triggered on S3 events (s3:ObjectCreated:, s3:ObjectRemoved:).

[riteshbehal007]

@vudh1 At your convenience would you mind giving us a brief update regarding this issue?

[riteshbehal007]

Do we have anything as an update or can change the user who has help us on this.

[dipakjain94]

Hi @vudh1, Do we have any update on this issue?

[Vrq]

@AllanZhengYP or @ajredniwja,

This issue still persists, could you please have a look?

[bcm-afalco]

Hi there. Same problem here. I Tried to use a throttling method to deal with maximum rate at a given time but does not work.

[marco-primiceri]

Hi all, we are having the same problem too. On a staging environment, when nothing else is happening, we can randomly get a "Rate exceeded" error on the first invalidation request.

[Vrq]

@awstools @AllanZhengYP @kuhe @jrencz @kellertk @comcalvi @eduardomourar

We would really appreciate a comment on this issue and escalating this internally, because it affects a large number of AWS users.

[jrencz]

Briefly

I tend to think it's rather an CloudFront API issue than JS SDK issue. Or that there's some coincidence of those requests being sent interlaced with some other and that's why it hits throttling (but that's a pure speculation)

@dipakjain94 have you tried passing maxRetries option? Does it help?

In detail

If there were too many paths to invalidate it should respond with BatchTooLarge I guess. So that's not it. If number of invalidations being processed was too high that'd be TooManyInvalidationsInProgress

It seems as if it was about endpoint being called too frequently. @dipakjain94 is the code you show called in some kind of loop maybe?

The error is retryable then SDK problem might be that it hasn't been retried instead it was finalized. Might because we don't know if it was retried or not. Or how many times. We may assume that it has been retried and that it was the default number of times.

In the code sample we see that service has been instantiated with no options, so maxRetries should be "calculated by the specific service object that the request is being made to":

https://github.com/aws/aws-sdk-js/blob/b0fd3156c162454a0bb644cca68349904cd2e52e/lib/config.js#L31-L34

So since it's not set on cloudfront service it's 3:

https://github.com/aws/aws-sdk-js/blob/b0fd3156c162454a0bb644cca68349904cd2e52e/lib/service.js#L169-L172

btw I see only one service that has this adjusted and it's dynamodb:

https://github.com/aws/aws-sdk-js/blob/b0fd3156c162454a0bb644cca68349904cd2e52e/lib/services/dynamodb.js#L41-L44

From what I see in the issue I can't tell if there were retries or not in reported case and I haven't searched for test confirming it works but if it didn't then I guess it would have already been noticed.

[Vrq]

@jrencz thank you for the feedback!

I don't believe we are hitting CloudFront limits because we receive this error when hitting no more than 15 single file invalidation requests in a row.

How can the maxRetries option be set?

[Vrq]

Hi @jrencz

This issue is still not solved.

Is there any way or place where we could escalate it? Or only a direct email to Jeff Bezos remains? :)

[ajredniwja]

Hey Guys apologies this issue fell out of queue and went unseen. With that being said I can take it up, Reading at the docs for rate limits/quotas, I dont feel the number you mention come near that but not certain https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html#limits-invalidations

The rate limit exceeded error indicates that the API calls to CloudFront have exceeded the max limit. The error and the stack trace doesnt indicate the error to be an SDK error.

To verify we can log all the calls made by Lambda and go from there. Retires can be logged by using logger :

aws.config.update({
  logger: console,
});

Have also reached out to the cloudfront team to have more insight.

[ajredniwja]

The default CreateInvalidation rate limit is 0.5.

{
      "Key Prefix": "Operation:CreateInvalidation,aws-account:",
      "Burst": 5,
      "Rate": 0.5
    }

The limits mentioned here https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html#limits-invalidations are on number of files in Invalidation requests, not rate of sending CreateInvalidation requests.

[david-malouf-vizio]

Found this while searching for an issue at work... best I can put together from the AWS case we have open, there are actually "burst limits" (that's actually the search term that led me here!). We were sending in no more than 70 Invalidations in an hour, but 10 (or so) of them were being requested in a single second. This seems to have triggered a "Rate exceeded" error.

Best I can tell, the solutions are:

• throttle how many Invalidations requests are sent per second
• ask AWS to increase some kind of "Invalidation request burst limit" for a given CloudFront Distribution

[LYTzeng]

FYI: In fact, there's a request rate limit for CloudFront CreateInvalidation API, though it is not documented publicly. The CreateInvalidation API rate limit is a hard limit (i.e. you cannot submit a limit increase request to raise the limit). I would suggest that you can either:

1. Like David mentioned above, try implementing some throttling or queuing in your code. Conservatively, the frequency recommended is >= 1 rps.
2. Increase the number of invalidation paths in one CreateInvalidation API call, and reduce the number of CreateInvalidation API call per second.
3. If your use case is to keep invalidating caches in a high frequency basis, consider leveraging CacheDisabled cache policy on specific Cache Behaviors.

[kellertk]

Hello everybody. Some internal AWS information was posted to this issue; I have removed this information for now. The comments from @LYTzeng are correct and you should follow this guidance in the future.

[marcelkottmann]

Hello @LYTzeng and @kellertk , I have written a small reverse engineering test script with sdk (typescript) where I invalidate a single path in sequence, wait X ms and repeat this a maximum of 30 times. In this test, I observe "Rate Exceeded" errors every time when X is less than 1600 ms. It would be good if such a major restriction were (officially) documented somewhere.