Verify slow crawling

[wernight]

I gave a list of about 300 proxies but set CONCURRENT_REQUESTS = 64. Still it seems that crawling is very slow (like 1 page every few seconds on average), much slower than not using any proxy at all. Of course DOWNLOAD_DELAY is low.

Looking at it, it seems that people should usually also increase CONCURRENT_REQUESTS_PER_DOMAIN in these cases (i.e. with a list of many possibly bad proxies), but even then it's still pretty slow.

[cortesimone]

👍 me too, any chance to have somebody from the dev team to have a look in to this issue? Thanks.