crusty
crusty copied to clipboard
Broad Web Crawler
Hello, let4be First of all I want to say it is really impressive what you have built, I am really amazed, so congratulations. Furthermore, I see that you wrote in...
Figure a way to auto-tune domain concurrency(there is a ~perfect N based on CPU and network bandwidth available) Will need some kind of graceful adaptive algo which will look at...
It's especially noticeable when using shorter living jobs `soft_timeout` = 30s `hard_timeout` = 60s throughput falls almost 3 times...