apalis icon indicating copy to clipboard operation
apalis copied to clipboard

Lock-free Postgres implementation

Open joshbenaron opened this issue 1 year ago • 2 comments

Hey - what's the timeline for Lock-free Postgres implementation?

Feel free to outline the rough problem which needs to be solved and we (Bundlr) might be able to solve, depending on time commitment

joshbenaron avatar Jul 15 '23 12:07 joshbenaron

The idea behind this implementation is inspired by Que https://github.com/que-rb/que/blob/9b264060d81f639d60bc9084ad6cc7dfb8812eba/lib/que/poller.rb#L60-L113

The full context is provided by the lib author here: https://gist.github.com/chanks/7585810

This improvement would hopefully get postgres to 10k/s jobs from the current 1.5k/s in Github CI. This should be straight forward porting and if combined with #126, then I expect huge performance boost.

geofmureithi avatar Jul 16 '23 04:07 geofmureithi

What are your thoughts @joshbenaron ?

geofmureithi avatar Jul 21 '23 10:07 geofmureithi