celery
Unrecoverable error in Celery worker: ReadOnlyError with Redis backend
Hello,
I'm experiencing an issue with my Celery worker intermittently shutting down due to a ReadOnlyError when trying to write to my Redis instance. Here's the error message I'm seeing:
[2023-06-22 12:38:52,599: CRITICAL/MainProcess] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
This seems to be happening when my Redis instance goes into a read-only mode during sync.
I am using the following setup:
- Python 3.8
- Celery 5.2.7
- Django-redis 5.2.0
My docker-compose.yml for Redis is as follows:
redis: image: redis:latest command: ["redis-server", "--maxmemory 300mb", "--maxmemory-policy allkeys-lru"] ports: - "6379:6379"
And my Celery configuration in Django settings is:
CELERY_BROKER_URL = 'redis://redis:6379/0' CELERY_RESULT_BACKEND = 'redis://redis:6379/0' CELERY_ACCEPT_CONTENT = ['application/json'] CELERY_RESULT_SERIALIZER = 'json' CELERY_TASK_SERIALIZER = 'json'
I'm currently mitigating the issue by configuring my docker-compose to restart the Celery worker when it stops, but this is not a long-term solution. Is there a way to handle this ReadOnlyError within Celery, or to prevent Redis from going into read-only mode in the first place?
Thank you in advance for your help and suggestions.
