[CRITICAL] Consumer encountered an unexpected error, IndexError: pop from empty list

[mhdzumair]

Issues

GitHub issues are for bugs. If you have questions, please ask them on the mailing list.

Checklist

• [x] Does your title concisely summarize the problem?
• [x] Did you include a minimal, reproducible example?
• [x] What OS are you using?
• [x] What version of Dramatiq are you using?
• [x] What did you do?
• [x] What did you expect would happen?
• [x] What happened?

What OS are you using?

Initially failed in oracle Linux instances in k8s, Then tested with Debian. The issue is recently occurred in Oracle Redis cluster. Its recent issue, not from beginning.

What version of Dramatiq are you using?

1.16.0

What did you do?

Running dramatiq api.task --process 1, I noticed an issue on prod today. I tried to replicate it from local with the same Redis cluster and got the same result. After that, I tried deleting all keys with dramatiq* in Redis, but the issue persisted. Finally, I used another Redis container, and the issue was resolved.

What did you expect would happen?

Work as expected.

What happened?

Error log,

[2024-02-23 07:53:14,316] [PID 29212] [Thread-5] [dramatiq.worker.ConsumerThread(default.DQ)] [CRITICAL] Consumer encountered an unexpected error.
Traceback (most recent call last):
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 344, in __next__
    data = self.message_cache.pop(0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5931, in __call__
    return client.evalsha(self.sha, len(keys), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5307, in evalsha
    return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5291, in _evalsha
    return self.execute_command(command, sha, numkeys, *keys_and_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 536, in execute_command
    return conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
           ^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 537, in <lambda>
    lambda: self._send_command_parse_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 513, in _send_command_parse_response
    return self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 553, in parse_response
    response = connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/connection.py", line 524, in read_response
    raise response
redis.exceptions.NoScriptError: No matching script. Please use EVAL.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/worker.py", line 264, in run
    for message in self.consumer:
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 363, in __next__
    self.message_cache = messages = self.broker.do_fetch(
                                    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 278, in do_dispatch
    self._max_unpack_size(),
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/dramatiq/brokers/redis.py", line 255, in _max_unpack_size
    cls._max_unpack_size_val = DEFAULT_LUA_MAX_STACK or self.scripts["maxstack"]()
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5936, in __call__
    self.sha = client.script_load(self.script)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/commands/core.py", line 5377, in script_load
    return self.execute_command("SCRIPT LOAD", script)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 536, in execute_command
    return conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
           ^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 537, in <lambda>
    lambda: self._send_command_parse_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 513, in _send_command_parse_response
    return self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/client.py", line 553, in parse_response
    response = connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dedsec/.local/share/virtualenvs/mediafusion-qrlDKCVP/lib/python3.11/site-packages/redis/connection.py", line 524, in read_response
    raise response
redis.exceptions.ResponseError: unknown command 'SCRIPT', with args beginning with: 'LOAD' '-- This file is a part of Dramatiq. -- -- Copyright (C) 2020 CLEARTYPE SRL <[email protected]> -- -- Dramatiq is free s' 
[2024-02-23 07:53:14,319] [PID 29212] [Thread-5] [dramatiq.worker.ConsumerThread(default.DQ)] [INFO] Restarting consumer in 3.00 seconds.