kombu icon indicating copy to clipboard operation
kombu copied to clipboard

Use durable=True by default everywhere

Open trianglesis opened this issue 6 months ago • 1 comments

Description

Trying the variant where all queues and exchanges are durable by default.

Discussed at issue 2237 RabbitMQ recommendation: DOC Related to the celery PR.

Conf: cat /etc/rabbitmq/rabbitmq.conf

# https://www.rabbitmq.com/docs/deprecated-features
deprecated_features.permit.transient_nonexcl_queues = false

A pretty simple fix just changed booleans to durable=True

Tested in my DEV environment. Now running at prod version.

No issues with tasks, queues, managing and pinging/inspecting queues and workers.

I didn't run bundled tests, however. Only relied upon my envs.

My celery conf
def celery_setup():
    # Get creds and hosts
    setup_celery_env()
    # Setup django project
    django.setup()
    #  The backend is specified via the backend argument to Celery
    global app
    if app is None:
        app = Celery('server',
                     broker=RabbitMQCreds.BROKER,
                     backend=backend,
                     )

    app.conf.timezone = 'UTC'
    app.conf.enable_utc = True
    # General config:
    app.conf.update(
        result_backend=result_backend,
        accept_content=['pickle'],
        task_serializer='pickle',
        result_serializer='pickle',
        result_extended=True,
        task_track_started=True,
        beat_scheduler='django_celery_beat.schedulers:DatabaseScheduler',
        database_engine_options={'pool_timeout': 90},
        worker_prefetch_multiplier=1,
        worker_concurrency=1,
        worker_timer_precision=1.0,
        broker_heartbeat=10.0,
        broker_heartbeat_checkrate=2.0,
    )

    app.autodiscover_tasks()
    app.conf.update(
        worker_max_memory_per_child=1024 * 100,
        worker_max_tasks_per_child=100,
        worker_proc_alive_timeout=5.0,
        task_acks_late=False,
        task_acks_on_failure_or_timeout=True,
        worker_cancel_long_running_tasks_on_connection_loss=True,
        task_reject_on_worker_lost=False,
        worker_pool_restarts=True,
        worker_enable_remote_control=True,
        worker_lost_wait=20,
        broker_connection_retry_on_startup=True,
        broker_connection_retry=True,
        broker_connection_max_retries=0,
        broker_connection_timeout=4.0,
        broker_channel_error_retry=True,
        broker_pool_limit=100,
        task_send_sent_event=True,
        worker_send_task_events=True,
        worker_disable_rate_limits=True,
        task_default_queue='[email protected]',
        task_default_exchange='someserver',
        task_default_routing_key='[email protected]',
        worker_direct=True,
        result_persistent=True,
        task_default_delivery_mode='persistent',
    )

trianglesis avatar Jun 10 '25 09:06 trianglesis

Thanks

auvipy avatar Jun 10 '25 12:06 auvipy

handled in another pr

auvipy avatar Sep 08 '25 04:09 auvipy