Alexander Sibiryakov
Alexander Sibiryakov
1. @RajatGoyal tell me about your desired RPS (can do it privately, or by jabber). 2. `_get_or_create_db_page(response)` seems to be a hot point. When several processes are first querying for...
Try sibiryakov [!!] scrapinghub.com
Hey @lljrsr long time no see ;) Try ``` bash $scrapy crawl [your spider] -s SPIDER_PARTITION_ID=[number] ``` it should work, because it's possible to configure Frontera using Scrapy settings (see...
Option values isn't passing. Well, can you investigate that? The same thing without -s FRONTERA_SETTINGS ?
It's connected with this https://github.com/scrapinghub/frontera/pull/105
https://github.com/scrapinghub/frontera/pull/110
Should be fine now. Please reopen in case of problems.
`FRONTERA_SETTINGS` module isn't connected with Scrapy anyhow, so Frontera's settings have precedence. http://frontera.readthedocs.org/en/latest/topics/scrapy-integration.html#frontier-scrapy-settings
It is in the docs: http://frontera.readthedocs.org/en/latest/topics/scrapy-integration.html#defining-frontier-settings-via-scrapy-settings Frontera is designed in a way to be used independently from Scrapy, so it happened historically Frontera has it's own settings. At the moment...
That would require talking directly with Scrapy Downloader, which isn't possible using Scrapy Scheduler interface.