dramatiq icon indicating copy to clipboard operation
dramatiq copied to clipboard

Worker randomly crashes while idle in k8s

Open vfronterotta opened this issue 1 year ago • 0 comments

Docker image: python:3.9.13-alpine3.16 Dramatiq version: 1.13

Using idle workers running in k8s pod connected to Redis broker. While testing locally everything seems to be working but when running in a k8s pod I am randomly getting the following error:

`[2022-09-07 21:34:08,057] [PID 9] [Thread-2] [dramatiq.worker.ConsumerThread(default)] [CRITICAL] Consumer encountered an unexpected error. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dramatiq/brokers/redis.py", line 344, in next data = self.message_cache.pop(0) IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dramatiq/worker.py", line 264, in run for message in self.consumer: File "/usr/local/lib/python3.9/site-packages/dramatiq/brokers/redis.py", line 363, in next self.message_cache = messages = self.broker.do_fetch( File "/usr/local/lib/python3.9/site-packages/dramatiq/brokers/redis.py", line 281, in do_dispatch return dispatch(args=args, keys=keys) File "/usr/local/lib/python3.9/site-packages/redis/commands/core.py", line 5704, in call return client.evalsha(self.sha, len(keys), *args) File "/usr/local/lib/python3.9/site-packages/redis/commands/core.py", line 5089, in evalsha return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args) File "/usr/local/lib/python3.9/site-packages/redis/commands/core.py", line 5073, in _evalsha return self.execute_command(command, sha, numkeys, *keys_and_args) File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1235, in execute_command return conn.retry.call_with_retry( File "/usr/local/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry return do() File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1236, in lambda: self._send_command_parse_response( File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1212, in _send_command_parse_response return self.parse_response(conn, command_name, **options) File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1251, in parse_response response = connection.read_response() File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 837, in read_response raise response redis.exceptions.ResponseError: Error running script (call to f_976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:128: @user_script: 128: The user executing the script can't run this command or subcommand`

It constantly recovers only to crash again a few minutes later.

vfronterotta avatar Sep 07 '22 21:09 vfronterotta