kombu
kombu copied to clipboard
Use durable=True by default everywhere
Description
Trying the variant where all queues and exchanges are durable by default.
Discussed at issue 2237
RabbitMQ recommendation: DOC
Related to the celery PR.
Conf: cat /etc/rabbitmq/rabbitmq.conf
# https://www.rabbitmq.com/docs/deprecated-features
deprecated_features.permit.transient_nonexcl_queues = false
A pretty simple fix just changed booleans to durable=True
Tested in my DEV environment. Now running at prod version.
No issues with tasks, queues, managing and pinging/inspecting queues and workers.
I didn't run bundled tests, however. Only relied upon my envs.
My celery conf
def celery_setup():
# Get creds and hosts
setup_celery_env()
# Setup django project
django.setup()
# The backend is specified via the backend argument to Celery
global app
if app is None:
app = Celery('server',
broker=RabbitMQCreds.BROKER,
backend=backend,
)
app.conf.timezone = 'UTC'
app.conf.enable_utc = True
# General config:
app.conf.update(
result_backend=result_backend,
accept_content=['pickle'],
task_serializer='pickle',
result_serializer='pickle',
result_extended=True,
task_track_started=True,
beat_scheduler='django_celery_beat.schedulers:DatabaseScheduler',
database_engine_options={'pool_timeout': 90},
worker_prefetch_multiplier=1,
worker_concurrency=1,
worker_timer_precision=1.0,
broker_heartbeat=10.0,
broker_heartbeat_checkrate=2.0,
)
app.autodiscover_tasks()
app.conf.update(
worker_max_memory_per_child=1024 * 100,
worker_max_tasks_per_child=100,
worker_proc_alive_timeout=5.0,
task_acks_late=False,
task_acks_on_failure_or_timeout=True,
worker_cancel_long_running_tasks_on_connection_loss=True,
task_reject_on_worker_lost=False,
worker_pool_restarts=True,
worker_enable_remote_control=True,
worker_lost_wait=20,
broker_connection_retry_on_startup=True,
broker_connection_retry=True,
broker_connection_max_retries=0,
broker_connection_timeout=4.0,
broker_channel_error_retry=True,
broker_pool_limit=100,
task_send_sent_event=True,
worker_send_task_events=True,
worker_disable_rate_limits=True,
task_default_queue='[email protected]',
task_default_exchange='someserver',
task_default_routing_key='[email protected]',
worker_direct=True,
result_persistent=True,
task_default_delivery_mode='persistent',
)
Thanks
handled in another pr