dockerfiles icon indicating copy to clipboard operation
dockerfiles copied to clipboard

Can't schedule spiders to work - scrapyd

Open ghost opened this issue 5 years ago • 2 comments

{"node_name": "4392081d8bcd", "status": "error", "message": "/usr/local/lib/python3.7/dist-packages/scrapy/utils/project.py:94: ScrapyDeprecationWarning: Use of environment variables prefixed with SCRAPY_ to override settings is deprecated. The following environment variables are currently defined: VERSION\n ScrapyDeprecationWarning\n"}

when I run 'curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider', the above is the output I get, is there a way to get around this?

ghost avatar Oct 28 '20 04:10 ghost

@ghost Did you manage to handle it ? Having the same issue...

miodeqqq avatar Dec 18 '21 17:12 miodeqqq

I've moved scrapyd to https://github.com/EasyPi/docker-scrapyd I'm running it without any problem.

version: "3.8"

services:

  scrapyd:
    image: easypi/scrapyd
    ports:
      - "6800:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - /usr/local/lib/python3.9/dist-packages
    restart: unless-stopped

vimagick avatar Dec 27 '21 09:12 vimagick