scaleable-crawler-with-docker-cluster icon indicating copy to clipboard operation
scaleable-crawler-with-docker-cluster copied to clipboard

use celery==4.0.2 can raise asyc error, celery==4.2.x won't. suggest to change celery version.

Open Vickey-Wu opened this issue 5 years ago • 1 comments

error as below:

worker_1    | /usr/local/lib/python2.7/site-packages/celery/platforms.py:793: RuntimeWarning: You're running the worker with superuser privileges: this is
worker_1    | absolutely not recommended!
worker_1    | 
worker_1    | Please specify a different user using the -u option.
worker_1    | 
worker_1    | User information: uid=0 euid=0 gid=0 egid=0
worker_1    | 
worker_1    |   uid=uid, euid=euid, gid=gid, egid=egid,
scaleable-crawler-with-docker-cluster_worker_2 exited with code 1
worker_1    | Traceback (most recent call last):
worker_1    |   File "/usr/local/bin/celery", line 10, in <module>
worker_1    |     sys.exit(main())
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
worker_1    |     _main()
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
worker_1    |     cmd.execute_from_commandline(argv)
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
worker_1    |     super(CeleryCommand, self).execute_from_commandline(argv)))
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
worker_1    |     return self.handle_argv(self.prog_name, argv[1:])
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 480, in handle_argv
worker_1    |     return self.execute(command, argv)
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 412, in execute
worker_1    |     ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 221, in run_from_argv
worker_1    |     return self(*args, **options)
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 244, in __call__
worker_1    |     ret = self.run(*args, **kwargs)
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 255, in run
worker_1    |     **kwargs)
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 99, in __init__
worker_1    |     self.setup_instance(**self.prepare_args(**kwargs))
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 122, in setup_instance
worker_1    |     self.should_use_eventloop() if use_eventloop is None
worker_1    |   File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 241, in should_use_eventloop
worker_1    |     self._conninfo.transport.implements.async and
worker_1    |   File "/usr/local/lib/python2.7/site-packages/kombu/transport/base.py", line 127, in __getattr__
worker_1    |     raise AttributeError(key)
worker_1    | AttributeError: async

Vickey-Wu avatar Mar 28 '19 07:03 Vickey-Wu

I updated the code here: https://github.com/brammittendorff/scaleable-crawler-with-docker-cluster

brammittendorff avatar Jun 17 '22 09:06 brammittendorff