celery icon indicating copy to clipboard operation
celery copied to clipboard
verified publisher icon celery

RabbitMQ detects cycle and drops message when using quorum queues with delayed tasks + retry

Open daniellang-dt opened this issue 4 months ago • 6 comments

Checklist

  • [x] I have verified that the issue exists against the main branch of Celery.
  • [ ] This has already been asked to the discussions forum first.
  • [x] I have read the relevant section in the contribution guide on reporting bugs.
  • [x] I have checked the issues list for similar or identical bug reports.
  • [x] I have checked the pull requests list for existing proposed fixes.
  • [x] I have checked the commit log to find out if the bug was already fixed in the main branch.
  • [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).
  • [ ] I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

  • [x] I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
  • [x] I have verified that the issue exists against the main branch of Celery.
  • [x] I have included the contents of pip freeze in the issue.
  • [x] I have included all the versions of all the external dependencies required to reproduce this bug.

Optional Debugging Information

  • [ ] I have tried reproducing the issue on more than one Python version and/or implementation.
  • [ ] I have tried reproducing the issue on more than one message broker and/or result backend.
  • [ ] I have tried reproducing the issue on more than one version of the message broker and/or result backend.
  • [ ] I have tried reproducing the issue on more than one operating system.
  • [ ] I have tried reproducing the issue on more than one workers pool.
  • [ ] I have tried reproducing the issue with autoscaling, retries, ETA/Countdown & rate limits disabled.
  • [ ] I have tried reproducing the issue after downgrading and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version: 5.5.3

celery report Output: 

software -> celery:5.5.3 (immunity) kombu:5.6.0b2 py:3.12.8
            billiard:4.2.1 py-amqp:5.3.1
platform -> system:Linux arch:64bit, ELF
            kernel version:6.8.0-51-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:disabled

deprecated_settings: None
broker_url: 'amqp://user:********@localhost:5672/my_host'
task_default_exchange_type: 'topic'
task_queues: [<unbound Queue celery-quorum -> <unbound Exchange celery-quorum(topic)> -> celery-quorum>]
task_default_queue: 'celery-quorum'
broker_transport_options: 
 'confirm_publish': True}

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: N/A or Unknown
  • Minimal Celery Version: N/A or Unknown
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: RabbitMQ 3.13.7 and 4.1.2
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output: 

amqp==5.3.1  
billiard==4.2.1  
boto3==1.40.13  
botocore==1.40.13  
celery @ git+https://github.com/celery/celery@33eb14852310996b1909c8388cd319809d6c8626  
certifi==2025.8.3  
charset-normalizer==3.4.3  
click==8.2.1  
click-didyoumean==0.3.1  
click-plugins==1.1.1.2  
click-repl==0.3.0  
debugpy==1.8.16  
docker==7.1.0  
idna==3.10  
iniconfig==2.1.0  
jmespath==1.0.1  
kombu==5.6.0b2  
packaging==25.0  
pluggy==1.6.0  
prompt-toolkit==3.0.51  
psutil==7.0.0  
pygments==2.19.2  
pytest==8.4.1  
pytest-celery==1.2.1  
pytest-docker-tools==3.1.9  
python-dateutil==2.9.0.post0  
python-dotenv==1.1.1  
python-memcached==1.62  
redis==6.4.0  
requests==2.32.5  
s3transfer==0.13.1  
setuptools==80.9.0  
six==1.17.0  
tenacity==9.1.2  
tzdata==2025.2  
urllib3==1.26.20  
vine==5.1.0  
wcwidth==0.2.13

Other Dependencies

N/A

Minimally Reproducible Test Case

# celeryconfig.py
RABBITMQ_USERNAME = os.getenv('RABBITMQ_USERENAME', 'guest')
RABBITMQ_PASSWORD = os.getenv('RABBITMQ_PASSWORD', 'guest')

app = Celery('celery_test')

app.conf.update(
    broker_url=f'pyamqp://{RABBITMQ_USERNAME}:{RABBITMQ_PASSWORD}@localhost:5672/my_vhost',
    task_default_exchange_type='topic',
    task_queues=[
        Queue(
            'celery-quorum', 
            exchange=Exchange('celery-quorum', type='topic'),
            routing_key='celery-quorum',
            queue_arguments={'x-queue-type': 'quorum'}
        ),
    ],
    task_default_queue='celery-quorum',
    broker_transport_options={"confirm_publish": True},
)

app.autodiscover_tasks(['tasks'])


# tasks.py
@app.task(autoretry_for=(ValueError,), retry_kwargs={"max_retries": 2},default_retry_delay=15)
def dummy_task():
    print("Dummy task is running")
    raise ValueError("Dummy task is broken")

# client.py
from tasks import dummy_task

dummy_task.apply_async(countdown=4)

Expected Behavior

The message should be retried twice and then discarded.

Actual Behavior

The message is dropped by rabbitmq, because it detects a cycle. That way it is stuck in the RabbitMQ queue as dead-lettered:

Image

This bug is an interplay between native delayed delivery, and an ETA/Countdown task that is retried. In the example given above, celery puts the task in delayed_delivery_2 for the 4s initial delay. Then it is executed by the worker and (because it fails and is retried) is sent to RabbitMQ again with all the previous headers. When it reaches the queue delayed_delivery_2 again during the retry, RabbitMQ will detect a cycle and drop it.

We investigated this quite a bit and this is probably, because celery copies all the headers when retrying a task. This includes the "X-Death" header, which apparently RabbitMQ uses to detect cycles. This header should probably stripped when retrying tasks in celery.

daniellang-dt avatar Aug 21 '25 13:08 daniellang-dt

Is anyone working on this ?

sashu2310 avatar Aug 26 '25 09:08 sashu2310

I don't think so.

auvipy avatar Aug 31 '25 09:08 auvipy

Not sure if this is relevant here but just copying here in case

logs from rabbitmq 4.1.4-management using celery quorum queues

2025-10-08 06:13:44.867536+00:00 [warning] <0.1439.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_0' in vhost 'my_host' because its configured dead-letter-exchange exchange 'celery_delayed_delivery' in vhost 'my_host' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again

Not sure if this is something i should handle on my side or part of celery functional requirements

David2011Hernandez avatar Oct 08 '25 06:10 David2011Hernandez

I'm not sure if this is relevant, but this commit documents how RabbitMQ interprets the x-death header: https://github.com/rabbitmq/rabbitmq-server/commit/320d8aeaf6506f5278284e5a850eb032b37bcbca

lukebakken avatar Nov 12 '25 16:11 lukebakken

I can reproduce this. (Thanks for the great minimal reproducer!) The death cycle here eventually causes disk space issues for RabbitMQ because the queues are configured with the at-least-once dead lettering strategy.

QQ storage background...

Quorum queues are replicated state machines that store a log of commands that modify the queue state. (Think "enqueue," "dequeue," "requeue," "ack," "nack," etc.) QQs track the commands that correspond to each message (those enqueue or requeue commands). When the oldest messages in the queue have been fully consumed and acknowledged, and then the QQ truncates away the old parts of the log with those commands. QQs have a concept of a "smallest raft index" which is the point in the log where the QQ can truncate up to. That's the smallest of oldest ready message, oldest message checked out to a consumer and oldest message in its internal dead letter queue when configured with at-least-once dead lettering.

So, when messages are stuck in the internal dead-letter queue because they can't be routed, QQs can't take snapshots and free disk space. When that happens you see those logs, for example from reproducing this:

2025-11-12 11:14:34.930313-05:00 [warning] <0.1654.0> Dead-letter queues cycle detected for source quorum queue 'celery_delayed_3' in vhost 'my_vhost' with dead-letter exchange exchange 'celery_delayed_2' in vhost 'my_vhost' and routing keys [undefined]: [<<"celery_delayed_2">>,
2025-11-12 11:14:34.930313-05:00 [warning] <0.1654.0>                                                                                                                                                                                                           <<"celery_delayed_3">>,
2025-11-12 11:14:34.930313-05:00 [warning] <0.1654.0>                                                                                                                                                                                                           <<"celery_delayed_2">>] This message will not be logged again.
2025-11-12 11:14:34.930468-05:00 [warning] <0.1654.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_3' in vhost 'my_vhost' with configured dead-letter-exchange exchange 'celery_delayed_2' in vhost 'my_vhost' and configured dead-letter-routing-key 'undefined'. This can happen either if the dead-letter routing topology is misconfigured (for example no queue bound to dead-letter-exchange or wrong dead-letter-routing-key configured) or if non-mirrored classic queues are bound whose host node is down. Fix this issue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.

The only way to free these dead-lettered messages that appear to be in a cycle is to route them to a new queue which is not already in the cycle. From the reproduction script I had a message stuck in celery_delayed_3. A rough outline of the steps in the management UI to move the stuck dead-letter message:

  1. Create a new queue, say my_dlx_sink
  2. Go to the exchange detail page for celery_delayed_2 (dead letter exchange of queue celery_delayed_3)
  3. Copy the routing key for the binding that leads to celery_delayed_2
  4. Unbind that binding
  5. Add a new binding that leads to queue my_dlx_sink using the copied routing key
  6. Wait for the QQ to retry routing the dead-lettered messages. celery_delayed_3's count of dead lettered messages should fall to zero after a few minutes and the QQ will clean up its disk space
  7. Go back to the celery_delayed_2 exchange, delete the binding to my_dlx_sink and add the original which pointed to the celery_delayed_2 queue

the-mikedavis avatar Nov 12 '25 17:11 the-mikedavis

@daniellang-dt did you reproduce this against 4.1.2? With your reproduction script I was unable to reproduce this against 4.2.0 while I could reliably reproduce this against 3.13.7. According to the deaths header release note that Luke linked, it looks like this should've been worked around in RabbitMQ in v4.0.1. In my test the celery logs showed an error-level log:

[2025-11-13 21:10:30,284: ERROR/ForkPoolWorker-64] Task tasks.dummy_task[2959e07b-c3bc-49ef-b1be-bb7fc441d3b3] raised unexpected: ValueError('Dummy task is broken')
Traceback (most recent call last):
  File "/nix/store/wf8wh6y8di6hrk8ljjz42nw8x73akadv-python3-3.13.7-env/lib/python3.13/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/wf8wh6y8di6hrk8ljjz42nw8x73akadv-python3-3.13.7-env/lib/python3.13/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/wf8wh6y8di6hrk8ljjz42nw8x73akadv-python3-3.13.7-env/lib/python3.13/site-packages/celery/app/autoretry.py", line 60, in run
    ret = task.retry(exc=exc, **retry_kwargs)
  File "/nix/store/wf8wh6y8di6hrk8ljjz42nw8x73akadv-python3-3.13.7-env/lib/python3.13/site-packages/celery/app/task.py", line 743, in retry
    raise_with_context(exc)
    ~~~~~~~~~~~~~~~~~~^^^^^
  File "/nix/store/wf8wh6y8di6hrk8ljjz42nw8x73akadv-python3-3.13.7-env/lib/python3.13/site-packages/celery/app/autoretry.py", line 38, in run
    return task._orig_run(*args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/src/rabbitmq/server/tasks.py", line 31, in dummy_task
    raise ValueError("Dummy task is broken")
ValueError: Dummy task is broken

And all celery_delayed_N queues were fully empty - nothing stuck in the internal dead-letter queues.

This is a little bit unrelated but to cut down on log noise I think Celery could declare the delayed queues in reverse order.

Current logs during declaration... 
2025-11-14 02:32:19.548655+00:00 [notice] <0.2938.0> queue 'celery-quorum' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:19.548955+00:00 [info] <0.2872.0> ra: started cluster %2F_celery-quorum with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery-quorum',rabbit@localhost}
2025-11-14 02:32:21.759198+00:00 [notice] <0.2976.0> queue 'celery_delayed_27' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:21.759402+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_27 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_27',rabbit@localhost}
2025-11-14 02:32:21.765176+00:00 [warning] <0.2977.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_27' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_26' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:22.416201+00:00 [notice] <0.2991.0> queue 'celery_delayed_26' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:22.416413+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_26 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_26',rabbit@localhost}
2025-11-14 02:32:22.421961+00:00 [warning] <0.2992.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_26' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_25' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:23.074055+00:00 [notice] <0.3005.0> queue 'celery_delayed_25' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:23.074259+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_25 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_25',rabbit@localhost}
2025-11-14 02:32:23.079949+00:00 [warning] <0.3006.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_25' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_24' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:23.728807+00:00 [notice] <0.3016.0> queue 'celery_delayed_24' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:23.729019+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_24 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_24',rabbit@localhost}
2025-11-14 02:32:23.735056+00:00 [warning] <0.3017.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_24' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_23' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:24.386675+00:00 [notice] <0.3033.0> queue 'celery_delayed_23' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:24.386885+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_23 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_23',rabbit@localhost}
2025-11-14 02:32:24.392587+00:00 [warning] <0.3034.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_23' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_22' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:25.045332+00:00 [notice] <0.3049.0> queue 'celery_delayed_22' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:25.045577+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_22 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_22',rabbit@localhost}
2025-11-14 02:32:25.051149+00:00 [warning] <0.3050.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_22' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_21' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:25.700036+00:00 [notice] <0.3058.0> queue 'celery_delayed_21' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:25.700247+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_21 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_21',rabbit@localhost}
2025-11-14 02:32:25.706056+00:00 [warning] <0.3059.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_21' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_20' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:26.356637+00:00 [notice] <0.3072.0> queue 'celery_delayed_20' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:26.356840+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_20 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_20',rabbit@localhost}
2025-11-14 02:32:26.362754+00:00 [warning] <0.3073.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_20' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_19' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:27.017632+00:00 [notice] <0.3084.0> queue 'celery_delayed_19' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:27.017844+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_19 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_19',rabbit@localhost}
2025-11-14 02:32:27.023523+00:00 [warning] <0.3085.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_19' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_18' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:27.674251+00:00 [notice] <0.3102.0> queue 'celery_delayed_18' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:27.674453+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_18 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_18',rabbit@localhost}
2025-11-14 02:32:27.680207+00:00 [warning] <0.3103.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_18' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_17' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:28.330311+00:00 [notice] <0.3118.0> queue 'celery_delayed_17' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:28.330530+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_17 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_17',rabbit@localhost}
2025-11-14 02:32:28.336105+00:00 [warning] <0.3119.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_17' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_16' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:28.986666+00:00 [notice] <0.3136.0> queue 'celery_delayed_16' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:28.986878+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_16 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_16',rabbit@localhost}
2025-11-14 02:32:28.992574+00:00 [warning] <0.3137.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_16' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_15' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:29.647201+00:00 [notice] <0.3179.0> queue 'celery_delayed_15' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:29.647443+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_15 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_15',rabbit@localhost}
2025-11-14 02:32:29.653051+00:00 [warning] <0.3181.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_15' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_14' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:30.306760+00:00 [notice] <0.3192.0> queue 'celery_delayed_14' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:30.306981+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_14 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_14',rabbit@localhost}
2025-11-14 02:32:30.312790+00:00 [warning] <0.3193.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_14' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_13' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:30.966331+00:00 [notice] <0.3210.0> queue 'celery_delayed_13' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:30.966540+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_13 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_13',rabbit@localhost}
2025-11-14 02:32:30.972003+00:00 [warning] <0.3211.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_13' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_12' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:31.624074+00:00 [notice] <0.3226.0> queue 'celery_delayed_12' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:31.624279+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_12 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_12',rabbit@localhost}
2025-11-14 02:32:31.629952+00:00 [warning] <0.3227.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_12' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_11' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:32.284907+00:00 [notice] <0.3241.0> queue 'celery_delayed_11' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:32.285123+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_11 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_11',rabbit@localhost}
2025-11-14 02:32:32.291224+00:00 [warning] <0.3242.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_11' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_10' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:32.944406+00:00 [notice] <0.3269.0> queue 'celery_delayed_10' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:32.944640+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_10 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_10',rabbit@localhost}
2025-11-14 02:32:32.950441+00:00 [warning] <0.3275.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_10' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_9' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:33.605000+00:00 [notice] <0.3289.0> queue 'celery_delayed_9' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:33.605201+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_9 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_9',rabbit@localhost}
2025-11-14 02:32:33.612079+00:00 [warning] <0.3290.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_9' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_8' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:34.274702+00:00 [notice] <0.3320.0> queue 'celery_delayed_8' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:34.274950+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_8 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_8',rabbit@localhost}
2025-11-14 02:32:34.280661+00:00 [warning] <0.3321.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_8' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_7' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:34.943514+00:00 [notice] <0.3341.0> queue 'celery_delayed_7' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:34.943717+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_7 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_7',rabbit@localhost}
2025-11-14 02:32:34.949947+00:00 [warning] <0.3346.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_7' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_6' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:35.610706+00:00 [notice] <0.3370.0> queue 'celery_delayed_6' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:35.610907+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_6 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_6',rabbit@localhost}
2025-11-14 02:32:35.616656+00:00 [warning] <0.3371.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_6' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_5' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:36.270448+00:00 [notice] <0.3391.0> queue 'celery_delayed_5' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:36.270675+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_5 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_5',rabbit@localhost}
2025-11-14 02:32:36.277254+00:00 [warning] <0.3392.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_5' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_4' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:36.932935+00:00 [notice] <0.3414.0> queue 'celery_delayed_4' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:36.933166+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_4 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_4',rabbit@localhost}
2025-11-14 02:32:36.938981+00:00 [warning] <0.3415.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_4' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_3' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:37.595021+00:00 [notice] <0.3438.0> queue 'celery_delayed_3' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:37.595258+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_3 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_3',rabbit@localhost}
2025-11-14 02:32:37.601060+00:00 [warning] <0.3439.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_3' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_2' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:38.254450+00:00 [notice] <0.3456.0> queue 'celery_delayed_2' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:38.254674+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_2 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_2',rabbit@localhost}
2025-11-14 02:32:38.260493+00:00 [warning] <0.3457.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_2' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_1' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:38.914457+00:00 [notice] <0.3484.0> queue 'celery_delayed_1' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:38.914687+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_1 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_1',rabbit@localhost}
2025-11-14 02:32:38.920424+00:00 [warning] <0.3487.0> Cannot forward any dead-letter messages from source quorum queue 'celery_delayed_1' in vhost '/' because its configured dead-letter-exchange exchange 'celery_delayed_0' in vhost '/' does not exist. Either create the configured dead-letter-exchange or re-configure the dead-letter-exchange policy for the source quorum queue to prevent dead-lettered messages from piling up in the source quorum queue. This message will not be logged again.
2025-11-14 02:32:39.574414+00:00 [notice] <0.3509.0> queue 'celery_delayed_0' in vhost '/': candidate -> leader in term: 1 machine version: 7, last applied 0
2025-11-14 02:32:39.574619+00:00 [info] <0.2967.0> ra: started cluster %2F_celery_delayed_0 with 1 servers. 0 servers failed to start: []. Leader: {'%2F_celery_delayed_0',rabbit@localhost}

The declaration order always creates a queue which has an unroutable DLQ, and then the DLQ is declared. If the queues were declared from zero to 27 instead of 27 to zero then these logs should not be printed.

the-mikedavis avatar Nov 14 '25 03:11 the-mikedavis