scrapy-proxies icon indicating copy to clipboard operation
scrapy-proxies copied to clipboard

Response never received errors

Open mmotti opened this issue 4 years ago • 0 comments

Does anyone else experience timeout errors, specifically immediately after redirects?

I've only set this up today, but specifically https://www.game.co.uk/en/hardware/xbox-series-x/?contentOnly=&inStockOnly=true&listerOnly=&pageSize=100

I can fetch it ok with Scrapy fetch, but if I try to use a spider that crawls the URL, I hit a 302 redirect and my crawl just completely errors out from that point with immediate failures "response never received". It's not long timeouts, it's literally just erroring immediately.

Please could somebody help me? I'm fairly new to this and I have no idea what the cause may be.

I'm using a pool of 10 http proxies on port 80

mmotti avatar Dec 19 '20 22:12 mmotti