solid_queue icon indicating copy to clipboard operation
solid_queue copied to clipboard

Cross-check my TODO list for a Python worker to work with SolidQueue

Open MRigal opened this issue 2 months ago • 1 comments

In my current job, we have an open-source monorepo featuring a steering website, doing the data modelling and more in Ruby, linked to ETL processes ran in Python. We currently use faktory to communicate and send jobs between the two languages, but its limitations brought us to consider a move to SolidQueue, including writing a Python library worker+client/enqueuer.

I've been through the SolidQueue code and did not encounter difficulties to understand the main concepts, but would appreciate a short cross-check of my TODO list / plan to make sure I don't oversee something. I could see that there was some interest in #286 and #679 for that.

Python library to interact with SolidQueue

Basic choices

  1. Not reimplementing everything, no scheduler, no real dispatcher, almost just a worker. A thin dispatcher layer for threads, signals, config and co is probably required.
  2. Also no recurring tasks support in a first time
  3. Recommend to use language-specific queues to not mix Python and Ruby jobs

Worker feature set

  1. Get jobs by queue
  2. Get all jobs probably not in a first time
  3. Feature to have a Process (or Interpreter in Python 3.14+) Pool for concurrency
  4. Support for threads maybe in a second time
  5. Sync and async support?
  6. Add retry logic mimicking ActiveRecord that SolidQueue is relying on?

Client/enqueue feature set

  1. Possibility to enqueue a job.
  2. Bulk enqueue capacity maybe later

MRigal avatar Nov 10 '25 16:11 MRigal

Just to drop the thoughts here: to have a minimal client to enqueue job from any language/framework, another possibility would be to have a thin postgrest-like interface calling a DB function from API call (though it shits the complexity to DB auth...). This POC is pretty nice, but hard to generalize to all DBs: https://pgqueuer.readthedocs.io/en/latest/postgrest.html

MRigal avatar Nov 11 '25 10:11 MRigal