Multiple queues, different `Q_CLUSTER` settings

[akampen]

First off, thanks for giving new life to django-q!

In the old repo there are several issues discussing maintaining multiple queues. We've had success with a qclusters command that looks like this:

class Command(BaseCommand):
    help = "Start multiple django-q clusters"

    def handle(self, *args, **options):
        for config in settings.Q_CLUSTER_CONFIG_LIST:
            q = Cluster(get_broker(config["name"]))
            q.start()

However, we really want our two queues to have different number of workers and different settings for retry/timeout, but this isn't possible because every Cluster uses the settings found in settings.Q_CLUSTER. It'd be really great if we could pass in a Q_CLUSTER-like dictionary to the Cluster constructor to adjust these values.

Thoughts? Would this be difficult to add?

[GDay]

I like this idea!

The main issue here is that currently all Q2 settings (except for the broker) are fetched directly from the settings through a separate class that gets called pretty much anywhere, without having to rely on the cluster itself. This means that if we were to simply add the options to the Cluster constructor, we would likely miss a few cases.

For example here: https://github.com/GDay/django-q2/blob/c1d4858f622c7b05c825cc1f9d39cef6243ae3dc/django_q/signing.py#L13-L21

Having said that, adding WORKERS and TIMEOUT to the constructor should be fairly straightforward as those are only used within the cluster. RETRY is also used in the brokers, so that's likely a bit more work.

I don't think the monitoring tools will like that change though. https://django-q2.readthedocs.io/en/master/monitor.html

[sinowood]

I have done similar things in my local branch. I don't think the monitoring tools will like the change too, but I don't use it for monitoring LOL.