scrapyd
scrapyd copied to clipboard
scrapy
Remove deprecated environment variables for Scrapy 2
/usr/local/lib/python3.7/site-packages/scrapy/utils/project.py:94: ScrapyDeprecationWarning: Use of environment variables prefixed with SCRAPY_ to override settings is deprecated. The following environment variables are currently defined: JOB, LOG_FILE, SLOT, SPIDER
These variables are defined in environ.py
Can anybody explain how to deal with it.Thanks! /usr/local/lib/python3.6/dist-packages/scrapy/utils/project.py:94: ScrapyDeprecationWarning: Use of environment variables prefixed with SCRAPY_ to override settings is deprecated. The following environment variables are currently defined: JOB, LOG_FILE, SLOT, SPIDER ScrapyDeprecationWarning
My reading of Scrapy's code is that it doesn't have any JOB, SLOT or SPIDER settings, so it is warning about environment variables that it doesn't even use and has never used... It does have a LOG_FILE setting: https://docs.scrapy.org/en/latest/topics/settings.html#log-file
Besides SCRAPY_LOG_FILE, SCRAPY_FEED_URI also causes a deprecation warning. Scrapy's FEED_URI setting is deprecated in favor of a FEEDS dict. https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEEDS
Scrapyd needs to programmatically communicate the LOG_FILE and FEEDS settings to Scrapy, but with the deprecation of these environment variables, Scrapy's only interface is a settings file.
I guess Scrapyd could write temporary settings files, but this seems like a hack.
I'ved opened https://github.com/scrapy/scrapy/issues/5498 to discuss.
With some refactoring, we can probably do this:
pass a settings argument to this invocation of scrapy.cmdline.execute
from https://github.com/scrapy/scrapy/issues/5498#issuecomment-1122659457