poolboy icon indicating copy to clipboard operation
poolboy copied to clipboard

Local Pool (manager) with Remote Workers + Configurable Worker collection

Open matyasmarkovics opened this issue 5 years ago • 3 comments

This PR should really be 2 as the title suggest. However, it was easier for me the to create it as single one. - Apologies! - I just wanted to get your thoughts on the changes first and find out whether you see these extensions useful. - I'm happy to take this further, e.g.: split the PR, etc.

Remote Workers

I use poolboy at work and needed to create a pool of processes similar to the rpc module implementation in Erlang/OTP. - rpc (or rex) is a single process, I needed a pool of them. - This led me to extend poolboy, so that (local) workers could be started by a supervisor on a remote node.

Worker Collection

In situations, where the worker is non-blocking, the pool performs better if the checkout, checkin mechanism is not used. However, it's still a good idea to distribute the work-load between workers, e.g.: simply relying on random distribution. - This led me to consider random-access data-structures, like array and tuple for the collection of workers. - I've added a module: poolboy_collection that encapsulates the data-structure, in which the workers are stored.

matyasmarkovics avatar Aug 20 '19 11:08 matyasmarkovics

There's a lot of changes here and I'm not sure when I'll have time to review this :/ I will try to find some time though.

Vagabond avatar Aug 04 '20 21:08 Vagabond

Hi @Vagabond, you are right this has become a mess. Sorry about that. I would be happy to rework these changes into manageable chunks (PRs). Please let me know if that would actually work for you ;)

matyasmarkovics avatar Aug 05 '20 07:08 matyasmarkovics

Hi again @Vagabond, I've started on spitting this PR into manageable pieces.

Here's the first chunk: https://github.com/devinus/poolboy/pull/140 - Still somewhat large but I think it's manageable.

Thanks!

matyasmarkovics avatar Aug 18 '20 07:08 matyasmarkovics