scrapy-deltafetch icon indicating copy to clipboard operation
scrapy-deltafetch copied to clipboard

Does this plugin support redirected url?

Open mtabbasi opened this issue 3 years ago • 1 comments

I ran this plugin for https://quotes.toscrape.com/ but It redirects every requests and this plugin doesn't work for redirects

mtabbasi avatar Aug 20 '21 18:08 mtabbasi

I get all HTTP 200 responses for quotestoscrape. Are you using quotesbot project?

But when I test for redirects I see it actually doesn't filter out properly. Because it checks for response.request here: https://github.com/scrapy-plugins/scrapy-deltafetch/blob/master/scrapy_deltafetch/middleware.py#L85 instead of response.meta['redirect_urls'] which should be proper value for redirected items.

So it won't work for redirects @mtabbasi

pawelmhm avatar Sep 17 '21 18:09 pawelmhm