Add per-org/user concurrency capacity support

[rauchy]

What type of PR is this? (check all applicable)

• [x] Feature

Description

This PR provides the ability to control how many queries a user or organization can run simultaneously. Here's the gist of how it works:

1. When a worker picks up a job, it examines all the currently running jobs for that queue and counts the amount of jobs currently being executed by the new job's user and org.
2. If the amount of currently executing jobs by that user or org exceeds the policy (the policy provided in this PR is a "no limit" policy, but custom policies can be implemented under dynamic_settings.py), the job is moved to a waiting list (i.e a side queue, not actively watched by any worker) for that user or org.
3. The worker continues to the next job in queue.
4. After a worker finishes working on a job, it looks for a waiting list for that user or org.
5. If there's a waiting list for that user or org, the first item is popped and pushed back to the original queue.

If a (non-horse) worker dies, it could result in a job stuck in the waiting list until another worker completes work for that user/org. This is handled by a periodic maintenance job that looks for any waiting lists without a trigger.

[konnectr]

@rauchy thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI, as well as updating merge conflicts?

We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.

[konnectr]

I like the concept of this PR. I think it's worth doing it sometime

[justinclift]

I wonder if this PR is mostly for when Redash is running in "multi-org" mode?

Not sure if it makes as much sense for a Redash installation for only has a single organisation.

[rauchy]

Heh, talk about a late response! Sorry about that, I was on a sort of sabbatical.

@konnectr if you think it's worth a shot, I can try rebasing with these changes.

@justinclift yeah, for single org installations this doesn't make much sense, but it still offers user-based capacities, which can be useful. (also, how you doing, buddy? 😀)