[Feat] Provide more details for 429 Rate limit reached

[tractorjuice]

Other APIs provide details within the 429 response that enables a calculation or even the details of when to retry. For example:

Groq:

Error: Error code: 429 - {'error': {'message': 'Rate limit reached for model llama3-70b-8192 in organization org_xxxxxxx on tokens per minute (TPM): Limit 7000, Used 0, Requested ~12903. Please try again in 50.597142857s. Visit https://console.groq.com/docs/rate-limits for more information.', 'type': 'tokens', 'code': 'rate_limit_exceeded'}}

[nickscamara]

Great idea.

[simonha9]

Would love to take this - I see from openapi spec a 429 can be returned from:

1. /scrape
2. /crawl
3. /search
4. /crawl/status

What are the rate limits for each of these endpoints? I see in rate-limiter.ts there is a crawlStatusRateLimiter - should there be something similar for the other endpoints or should the 429 be passed-through from the underlying services like supabase auth or serp?

[nickscamara]

@simonha9 Yey! Go for it! Yep, so we should be pushing this week/next week a more strict enforcement on the rate limit based on the subscription. Right now crawlStatus is the only one that differs outside of the preview ones/ global ones.

[simonha9]

Sounds good! Please link a wiki or update this thread when you have an idea what the rate limits should be per subscription + if you have any specific verbiage in mind for the rate limit error messages and I can go ahead an implement this.

[nickscamara]

@simonha9 sounds good, will do! Thank you so much.

ccing firecrawl team: #127 should be done first before #81

[rafaelsideguide]

@simonha9 we've already implemented the rate limits as detailed in issue #151 . You can proceed with implementing the additional details for the 429 rate limit responses as discussed. Looking forward to seeing your contributions!

[simonha9]

Thanks for implementing the rate limits, I will start on this today 😄