solid_queue icon indicating copy to clipboard operation
solid_queue copied to clipboard

Link `claimed_executions` to `processes` via `process_name`

Open rosa opened this issue 5 months ago • 0 comments

That is, instead of linking them via the standard process_id (claimed_execution.process_id and process.id).

The reason for this change is that we're exploring the use of Solid Queue in a multi-tenanted setup, where we'll have an individual database per tenant, but a shared supervisor, dispatcher, scheduler, and workers for multiple tenants. We'll keep a supervisor-local database separated from the tenanted DBs, with just the solid_queue_processes table. The supervisor will control the processes stored in that table in that local DB, whereas each tenant will use its own database with the remaining tables, which will contain the jobs and the job metadata.

The consequence of this is that we can no longer rely on finding a process's in-flight jobs via its id, because the id can be (and most likely will be) duplicated across supervisors' local databases. The name, however, is randomly generated via SecureRandom.hex(10), which, with a very high likelihood, guarantees unique names across supervisors' local DBs. In this way, if a process is killed, leaving in-flight jobs (claimed executions) behind, and we fail over a different supervisor with a different local DB, that supervisor will be able to know which claimed executions are orphaned.

This requires a new migration, which is added here, but it's not enforced. The code still works without running the new migration, it just emits a deprecation warning like this:

DEPRECATION WARNING: Solid Queue has pending database migrations. To get the new migration files, run:
  rails solid_queue:update
And then:
  rails db:migrate
These migrations will be required after version 2.0

This also includes a new solid_queue:update command to facilitate installing this migration and future migrations.

cc @flavorjones

rosa avatar Jul 21 '25 13:07 rosa