riverqueue
River leader in multi-service ecosystem
Hey folks, I've the following use case and I would like to hear your opinion on a limitation I faced once starting to tackle it. I'm dealing with microservices architecture where N services would like to schedule periodic jobs (independently). The services are compute distributed with centralized single DB instance (multi schema). Meaning - each service creates a river client and set its internal scheduling jobs (worth to mention, each service defines its scheduling job kind). In this scenario with the current leader election process there is a race between the services on the leadership ownership. Let's assume service N_1 assigned to be the leader, as a result services N_2....,k schedule jobs will be starved (and actually won't run). Are you aware of this limitation? There is an alternative to achieve the requirements described aforementioned? Thanks in advanced.
I'm assuming the different microservice workers wouldn't know how to work each others jobs would they? Do you get around this by using separate queues?
@bgentry I suppose we could address this by exposing the river_leader.name field and making it a per-client configurable value so that different clients could be part of separate leadership races. This wouldn't be perfect though, because it would mean other maintenance services like the job cleaner would end up running in multiple places (that may be okay, but it might cause contention).
Are each of your services running on their own independent Postgres schema/namespace? I had always envisioned there being just a single leader in a given namespace, so if you want to have isolated services w/ independent jobs they would live on separate schemas and thus be able to have their own leader and their own maintenance processes for their own respective schema.
Our multi schema support is still held back by sqlc not supporting schemas other than as a connection param, but once we get past that we should be in a better place.
Thanks both @brandur and @bgentry. There is a chance to proceeded with @brandur solution as a midterm bypass to the obstacle? I can understand why its interfere the singularity of the rescue/cleanup jobs, but still it will give a significant granularity to the program to choose if to use a "global" leader or a dedicated "specific" leader.
We just ran into this issue as well. We have many apps that need to maintain their own jobs, queues and workers, but there is only one shared db schema. I don't pretend to understand the inner workings of river, but it would really great if we could use river like this. For us, each job is unique to the app that created it, and requiring access to resources only available in that app. Using different queues doesn't seem to solve the problem, and moving all workers to a common app (so most apps are just insert only) is not a viable solution for us.
The docs are vague about what the leader elector does, but implies that it's maintenance and cleanup related. Can't the client schedule and work jobs just for the queue(s) it's configured for, regardless of which instance is the leader? We see the queues get created and jobs inserted, but no jobs are ever started except in the app type that holds the leader.
Update on this: we are quite close to multi-schema support in #848, which I think will cover some cases like this.
The docs are vague about what the leader elector does, but implies that it's maintenance and cleanup related. Can't the client schedule and work jobs just for the queue(s) it's configured for, regardless of which instance is the leader? We see the queues get created and jobs inserted, but no jobs are ever started except in the app type that holds the leader.
The trouble is that jobs aren't tagged back to their client of origin, so you'd have multiple maintenance services like the job cleaner that end up targeting the same roles and contending with each other.
I suppose we could make each client's leadership name configurable, and either accept contention, let some maintenance services be disabled by changing their run internal to never, or something like that.