celery
'Supervisor' crashed
I have used these packages in my project:
celery==3.1.18 django-celery==3.1.16 kombu==3.0.26
And I started workers with option "--autoscale=50,15" I have many background tasks, after some mins, workers are crashed and I can see warning and errors in log:
[2015-05-21 02:48:35,093: ERROR/MainProcess] Process 'Worker-31' pid:20885 exited with 'exitcode None'
[2015-05-21 02:48:35,572: WARNING/MainProcess] 'Autoscaler' crashed: AssertionError() [2015-05-21 02:48:35,659: WARNING/MainProcess] Traceback (most recent call last): [2015-05-21 02:48:35,659: WARNING/MainProcess] File "/usr/local/lib/python2.7/dist-packages/celery/utils/threads.py", line 64, in run [2015-05-21 02:48:35,660: WARNING/MainProcess] body() [2015-05-21 02:48:35,660: WARNING/MainProcess] File "/usr/local/lib/python2.7/dist-packages/celery/worker/autoscale.py", line 81, in body [2015-05-21 02:48:35,661: WARNING/MainProcess] self.maybe_scale()
[2015-05-21 02:48:35,655: ERROR/MainProcess] Thread 'Supervisor' crashed: AssertionError() Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 474, in run return self.body() File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 533, in body pool._maintain_pool() File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1248, in _maintain_pool self._repopulate_pool(joined) File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1233, in _repopulate_pool self._create_worker_process(self._avail_index()) File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1237, in _avail_index assert len(self._pool) < self._processes AssertionError
I do not know the reason why workers are crashed and how to fix that ? could you please help me on this?
Thank you very much!
Also seen with
celery==3.1.18 kombu==3.0.30 billiard==3.3.0.20
02:21:34.907 +0000 ERROR [pool.error] Thread 'Supervisor' crashed: AssertionError()
Traceback (most recent call last):
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 474, in run
return self.body()
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 533, in body
pool._maintain_pool()
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1248, in _maintain_pool
self._repopulate_pool(joined)
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1233, in _repopulate_pool
self._create_worker_process(self._avail_index())
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1237, in _avail_index
assert len(self._pool) < self._processes
AssertionError
--
2017-06-06 05:59:39.920 +0000 ERROR [pool.error] Thread 'Supervisor' crashed: AssertionError()
Traceback (most recent call last):
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 474, in run
return self.body()
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 533, in body
pool._maintain_pool()
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1248, in _maintain_pool
self._repopulate_pool(joined)
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1233, in _repopulate_pool
self._create_worker_process(self._avail_index())
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1237, in _avail_index
assert len(self._pool) < self._processes
AssertionError
2017-06-06 05:59:39.920 +0000 ERROR [pool.error] Thread 'Supervisor' crashed: AssertionError()
Traceback (most recent call last):
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 474, in run
return self.body()
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 533, in body
pool._maintain_pool()
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1248, in _maintain_pool
self._repopulate_pool(joined)
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1233, in _repopulate_pool
self._create_worker_process(self._avail_index())
File "venv/lib/python2.7/site-packages/billiard/pool.py", line 1237, in _avail_index
assert len(self._pool) < self._processes
AssertionError
