[💡 FEATURE REQUEST]: Multiple pool support

[Kaspiman]

Plugin

JOBS

I have an idea!

We in the company actively use JOBS plugin and process a lot of tasks.

Our monolithic application has very different queue handlers with:

• a very long locks and small amount of tasks
• a very short duration and huge amount of tasks
• a regular and predictable number of tasks
• an unpredictable task spikes
• memory leaks

In the current JOBS implementation, all tasks added into one queue and processed by one pool of workers. Different queue handlers can greatly interfere with others. Problem handlers can take too many workers, requeue tasks too often, and take too long to process.

As a result, the JOBS-cluster performance is unstable and low.

I would like to be able to configure individual pools and assign them individual settings:

• the most important queues will always be processed in a separate pools
• long tasks will run separately from fast ones
• handlers with memory leaks will be restarted more often then non-leaked
• other problematic and experimental queues will be processed regardless of stable

The new feature proposes:

• configuring many pools
• start/destroy pools in parallel
• restart certain pool in case of network error
• collect metrics from each pool separately (maybe add tag with pool name)
• manage each pool via console commands

Here is an example config for solving these problems:

jobs:
    pools:
        predictable_pool:
            pool:
                num_workers: 80
                max_jobs: 10000
            consume:
                - predictable_queue_1
                - predictable_queue_2
            pipelines:
                predictable_queue_1:
                    driver: amqp
                    config:
                        queue: predictable_queue_1
                        prefetch: 40
                        # ...
                predictable-queue-2:
                    driver: amqp
                    config:
                        queue: predictable_queue_2
                        prefetch: 40
                        # ...
        slow_pool:
            pool:
                num_workers: 25
                max_jobs: 100 # restart more often
            consume:
                - leaked_queue_1
            pipelines:
                # ...

The result can be a mechanism for managing many pools in a each plugin. This mechanism can be reused in the future plugins.

Comrades, what do you think of this?

[rustatian]

Hey @Kaspiman 👋🏻 Thanks for the great idea 👍🏻 I'll plant it (initially) for the 2024.0 milestone (2023.3 is full now).