dragonflydb
BullMQ / Cached IP from previous Dragonfly leader
Describe the bug Not sure if this issue is with Dragonfly, Dragonfly Operator or BullMQ, I use Immich which has Dragonfly as a backend deployed by DragonFly Operator.
To Reproduce I have no issues until there is a leader change in Dragonfly (upgrade, node restart, etc) and Immich seems to be using a cached IP of the previous leader. The Dragonfly logs are full of:
W20250421 11:44:22.123201 12 main_service.cc:1374] EVALSHA b4cd4bbdf096cd8d06246080315ae81c56e05a46 10 immich_bull:library:wait immich_bull:library:active immich_bull:library:prioritized immich_bull:library:events immich_bull:library:stalled immich_bull:library:limiter immich_bull:library:delayed immich_bull:library:paused immich_bull:library:meta immich_bull:library:pc immich_bull:library: 1745235862122 \xde\x00\x03\xa5token\xd9*b204d227-a63d-4442-8c3a-d5b2621590b3:35140\xaclockDuration\xcdu0\xa7limiter\xc0 failed with reason: Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:36.187906 11 script_mgr.cc:328] Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:36.188086 11 main_service.cc:1374] EVALSHA b4cd4bbdf096cd8d06246080315ae81c56e05a46 10 immich_bull:search:wait immich_bull:search:active immich_bull:search:prioritized immich_bull:search:events immich_bull:search:stalled immich_bull:search:limiter immich_bull:search:delayed immich_bull:search:paused immich_bull:search:meta immich_bull:search:pc immich_bull:search: 1745235876187 \xde\x00\x03\xa5token\xd9*538e32b7-302a-4663-8ed0-6fa493e04e40:35137\xaclockDuration\xcdu0\xa7limiter\xc0 failed with reason: Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:37.226603 12 script_mgr.cc:328] Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:37.226760 12 main_service.cc:1374] EVALSHA b4cd4bbdf096cd8d06246080315ae81c56e05a46 10 immich_bull:library:wait immich_bull:library:active immich_bull:library:prioritized immich_bull:library:events immich_bull:library:stalled immich_bull:library:limiter immich_bull:library:delayed immich_bull:library:paused immich_bull:library:meta immich_bull:library:pc immich_bull:library: 1745235877225 \xde\x00\x03\xa5token\xd9*b204d227-a63d-4442-8c3a-d5b2621590b3:35140\xaclockDuration\xcdu0\xa7limiter\xc0 failed with reason: Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:40.165977 11 script_mgr.cc:328] Error running script (call to b8c8950186e870a4dfcc57e460f7637cd1c25077): @user_script:195: -READONLY You can't write against a read only replica.
W20250421 11:44:40.166092 11 main_service.cc:1374] EVALSHA b8c8950186e870a4dfcc57e460f7637cd1c25077 8 immich_bull:search:stalled immich_bull:search:wait immich_bull:search:active immich_bull:search:failed immich_bull:search:stalled-check immich_bull:search:meta immich_bull:search:paused immich_bull:search:events 1 immich_bull:search: 1745235880165 30000 failed with reason: Error running script (call to b8c8950186e870a4dfcc57e460f7637cd1c25077): @user_script:195: -READONLY You can't write against a read only replica.
W20250421 11:44:40.517180 11 script_mgr.cc:328] Error running script (call to b8c8950186e870a4dfcc57e460f7637cd1c25077): @user_script:195: -READONLY You can't write against a read only replica.
I reached out to Immich project and they suspect it is an issue with BullMQ library. I'm an end-user, not a developer, just reporting what I see.
Expected behavior Dragonfly / BullMQ library should be aware of new IP address after leader change.
Environment (please complete the following information):
- OS: Ubuntu 24.04.02
- Kernel: 6.8.0-57-generic
- Containerized?: Kubernetes
- Dragonfly Version: 1.29.0
Reproducible Code Snippet
# Minimal code snippet to reproduce this bug
Additional context Add any other context about the problem here.
Hi, it looks like your app tries to execute write commands against the dragonfly replica. If you are using dragonfly operator I believe @Pothulapati can help with this
Hey @reefland, The Kuberentes Operator updates the Service (i.e Load Balancer) to use a different master.So, how are you using the endpoint of Dragonfly here? like the service IP or DNS name?
Hey @reefland, The Kuberentes Operator updates the Service (i.e Load Balancer) to use a different master.So, how are you using the endpoint of Dragonfly here? like the service IP or DNS name?
I'm using the DNS names within Kubernetes.
$ k get svc -n dragonfly 130 ↵
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dragonfly ClusterIP 10.43.30.25 <none> 6379/TCP 290d
dragonfly-dragonfly-operator ClusterIP 10.43.59.11 <none> 8081/TCP,8080/TCP 290d
$ nslookup dragonfly.dragonfly.svc.cluster.local
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local
Name: dragonfly.dragonfly.svc.cluster.local
Address 1: 10.43.30.25 dragonfly.dragonfly.svc.cluster.local
And then Immich is configured to use the DNS name:
REDIS_HOSTNAME: dragonfly.dragonfly.svc.cluster.local
REDIS_PORT: "6379"
REDIS_DBINDEX: "0"