429 rate limit of 1 request per minute

[monkeybiz55]

trafficstars

I switched to the latest docker image and started to see the 429 error code, when crawl4ai is saying rate limit of request is 1 per minute. However, such rate limit wasn't there when I build from source from earlier versions. Question:

1. Is this 1 request / minute rate limit from docker host a must for the program to perform?
2. If not, how do i modify this rate limit when using docker? Much appreciated!

[unclecode]

I suggest you go to the root of the projects and edit the main.py file. There, you will see that we applied the rate limit in different ways. If you see some variables, we read from environment variables like ACCESS_PER_MIN and ACCESS_TIME_SPAN, and they control this kind of limit. However, you can disable them, or a better way is to look at it, as we already accept an access token. So, you can set your own access token, then set the access token in your environment variable. And whenever you pass the access token, all these limits will be disabled. That's your choice. Nevertheless, where you can do that is in the main file. Let me know if you encounter any difficulties, and I will be happy to help you.

[monkeybiz55]

thanks unclecode, just wasn't sure where you put the rate limit params. i can take it from here. btw great job for the async update, was gonna ask for it. you are on fire!

[unclecode]

@monkeybiz55 Thank you so much and for your nice words. I'm glad to know that it was helpful. Let me know if you need any help for this rate limit. Happy crawling