Distributed-Multi-User-Scrapy-System-with-a-Web-UI icon indicating copy to clipboard operation
Distributed-Multi-User-Scrapy-System-with-a-Web-UI copied to clipboard

Worker hosts are unreachable

Open sap9433 opened this issue 6 years ago • 0 comments

I started scrapyd , mongo and rabbitmq in worker threads still it shows worker status: unreachable , version: unknown

When I cal ping workers from Link generator curl working file

This is how ./scrapyproject/scrapy_packages/settings.py looks

`SCHEDULER = ".rabbitmq.scheduler.Scheduler" SCHEDULER_PERSIST = True RABBITMQ_HOST = '127.0.0.1' RABBITMQ_PORT = 5672 RABBITMQ_USERNAME = 'guest' RABBITMQ_PASSWORD = 'guest'

MONGODB_PUBLIC_ADDRESS = '127.0.0.1:27017' # This will be shown on the web intee rface, but won't be used for connecting to DB MONGODB_URI = '127.0.0.1:27017' # Actual uri to connect to DB MONGODB_USER = '' MONGODB_PASSWORD = '' MONGODB_SHARDED = False MONGODB_BUFFER_DATA = 100

LINK_GENERATOR = 'http://127.0.0.1:6800' # Set your link generator worker addree ss here SCRAPERS = ['172.17.0.4:3800', '172.17.0.2:6800'] # Set your scraper worker addd resses here

LINUX_USER_CREATION_ENABLED = False # Set this to True if you want a linux userr account created during registration`

And This is what I get when I curl them from link generator -

`root@96f3b9b83573:/usr/local/Distributed-Multi-User-Scrapy-System-with-a-Web-UI# curl -I 172.17.0.4:3800 HTTP/1.1 200 OK Date: Fri, 11 May 2018 04:50:31 GMT Content-Length: 699 Content-Type: text/html Server: TwistedWeb/18.4.0

root@96f3b9b83573:/usr/local/Distributed-Multi-User-Scrapy-System-with-a-Web-UI# curl -I 172.17.0.2:6800 HTTP/1.1 200 OK Date: Fri, 11 May 2018 04:50:50 GMT Content-Length: 699 Content-Type: text/html Server: TwistedWeb/18.4.0`

So workers are up .. but somehow link generator cant connect to them.

sap9433 avatar May 11 '18 04:05 sap9433