scaleable-crawler-with-docker-cluster
scaleable-crawler-with-docker-cluster copied to clipboard
use celery==4.0.2 can raise asyc error, celery==4.2.x won't. suggest to change celery version.
error as below:
worker_1 | /usr/local/lib/python2.7/site-packages/celery/platforms.py:793: RuntimeWarning: You're running the worker with superuser privileges: this is
worker_1 | absolutely not recommended!
worker_1 |
worker_1 | Please specify a different user using the -u option.
worker_1 |
worker_1 | User information: uid=0 euid=0 gid=0 egid=0
worker_1 |
worker_1 | uid=uid, euid=euid, gid=gid, egid=egid,
scaleable-crawler-with-docker-cluster_worker_2 exited with code 1
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/bin/celery", line 10, in <module>
worker_1 | sys.exit(main())
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
worker_1 | _main()
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
worker_1 | cmd.execute_from_commandline(argv)
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
worker_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
worker_1 | return self.handle_argv(self.prog_name, argv[1:])
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 480, in handle_argv
worker_1 | return self.execute(command, argv)
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 412, in execute
worker_1 | ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 221, in run_from_argv
worker_1 | return self(*args, **options)
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 244, in __call__
worker_1 | ret = self.run(*args, **kwargs)
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 255, in run
worker_1 | **kwargs)
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 99, in __init__
worker_1 | self.setup_instance(**self.prepare_args(**kwargs))
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 122, in setup_instance
worker_1 | self.should_use_eventloop() if use_eventloop is None
worker_1 | File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 241, in should_use_eventloop
worker_1 | self._conninfo.transport.implements.async and
worker_1 | File "/usr/local/lib/python2.7/site-packages/kombu/transport/base.py", line 127, in __getattr__
worker_1 | raise AttributeError(key)
worker_1 | AttributeError: async
I updated the code here: https://github.com/brammittendorff/scaleable-crawler-with-docker-cluster