Rate limits on nftstorage.link OPTIONS requests

[zpalmtree]

Hi, I have an app which is often hitting this domain in browser.

On occasion I hit rate limits, I am working on improving my software to handle this better, however, it seems the domain will also return a 429 on a browser OPTION preflight request.

Due to how CORS requests work, I can't get the error code programmatically in browser to determine that it was a 429 and stop sending requests to the server. https://stackoverflow.com/a/47418538/8737306

Is it possible to only apply 429s on regular requests?

It would also be nice to have the rate limit apply over a shorter period, with shorter limits, to make it easier to implement a generic exponential backoff algorithm on hitting a rate limit, rather than needing a more custom solution explicitly for this domain, since 30 seconds is a bit of an unacceptable delay on loading speeds.

Thanks.

[vasco-santos]

Hey @zpalmtree

Thanks for reaching out. I totally agree we should not have rate limiting for OPTIONS requests and that should not be blocking. Can you confirm?

It would also be nice to have the rate limit apply over a shorter period, with shorter limits, to make it easier to implement a generic exponential backoff algorithm on hitting a rate limit, rather than needing a more custom solution explicitly for this domain, since 30 seconds is a bit of an unacceptable delay on loading speeds.

Could you open an issue for this please? Probably an example on how that would benefit users would be rad to try to prioritise this change

[zpalmtree]

Sorry for the slow response.

I do seem to still be seeing CORS related issues when getting rate limited.

It appears to be Cloudflare enforcing the rate limits:

Maybe it's not that OPTIONS requests are getting rate limited, but there is no Access-Control-Allow-Origin header present when a 429 is returned on a regular GET?

It seems when that header is not present, fetch() in browser will reject the promise and return a generic error, with no way for me to detect it was a 429 causing it.

Indeed, looking at the headers of one of my 429 responses, it is missing Access-Control-Allow-Origin:

Then requesting the file later when the rate limit has elapsed, the header is present:

[vasco-santos]

When you get rate limited, you will likely be for any kind of following request, given the client will be blocked for 1min. I could not replicate locally to get 200 options requests being rate limited.

From app standpoint, the recommendation would really be to lazy load while user is scrolling rather than getting all resources at the same time

[vasco-santos]

Closing. Feel free to re-open if still an issue