scrapy-redis icon indicating copy to clipboard operation
scrapy-redis copied to clipboard

Logs report a warning: Spider.make_requests_from_url method is deprecated...

Open LittleHello opened this issue 4 years ago • 2 comments

Environments : scrapy-redis 0.6.8, Scrapy 2.4.1, Python 3.8.5 When running the spider, the logs report a warning : Spider.make_requests_from_url method is deprecated: it will be removed and not be called by the default Spider.start_requests method in future Scrapy releases. Please override Spider.start_requests method instead. Warning is from line 66 in https://github.com/scrapy/scrapy/blob/master/scrapy/spiders/init.py, the reason may be in https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/spiders.py, method named make_request_from_data in line 121. Can this warning be fixed?

LittleHello avatar Dec 11 '20 05:12 LittleHello

use make_request_from_data function and override it;

nieweiming avatar Mar 25 '21 01:03 nieweiming

As @nieweiming said, you can avoid the warning for your project by overriding make_request_from_data and instead of calling make_requests_from_url just create the Request object directly.

That could be a good PR too.

rmax avatar Mar 25 '21 09:03 rmax