dramatiq
dramatiq copied to clipboard
Worker randomly crashes while idle in k8s
Docker image: python:3.9.13-alpine3.16 Dramatiq version: 1.13
Using idle workers running in k8s pod connected to Redis broker. While testing locally everything seems to be working but when running in a k8s pod I am randomly getting the following error:
`[2022-09-07 21:34:08,057] [PID 9] [Thread-2] [dramatiq.worker.ConsumerThread(default)] [CRITICAL] Consumer encountered an unexpected error. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dramatiq/brokers/redis.py", line 344, in next data = self.message_cache.pop(0) IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/dramatiq/worker.py", line 264, in run
for message in self.consumer:
File "/usr/local/lib/python3.9/site-packages/dramatiq/brokers/redis.py", line 363, in next
self.message_cache = messages = self.broker.do_fetch(
File "/usr/local/lib/python3.9/site-packages/dramatiq/brokers/redis.py", line 281, in do_dispatch
return dispatch(args=args, keys=keys)
File "/usr/local/lib/python3.9/site-packages/redis/commands/core.py", line 5704, in call
return client.evalsha(self.sha, len(keys), *args)
File "/usr/local/lib/python3.9/site-packages/redis/commands/core.py", line 5089, in evalsha
return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args)
File "/usr/local/lib/python3.9/site-packages/redis/commands/core.py", line 5073, in _evalsha
return self.execute_command(command, sha, numkeys, *keys_and_args)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1235, in execute_command
return conn.retry.call_with_retry(
File "/usr/local/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1236, in
It constantly recovers only to crash again a few minutes later.