datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

Ballista cluster sharing by multiple users (with fairness of resource allocation)

Open gopik opened this issue 3 years ago • 0 comments

Is your feature request related to a problem or challenge? Please describe what you are trying to do. Let's say we've an organization that deploys ballista cluster to which multiple users connect to. It'd be great if executor slots can be allocated to the users fairly. If a query has consumed all slots allotted to a user, it can't progress until more slots are generated (say by passage of time).

Describe the solution you'd like Each query execution is associated with some identity, identities are rate limited for the slots and tasks for the identities are queued and consumed at allowed rate (or concurrency)

Describe alternatives you've considered

Additional context Add any other context or screenshots about the feature request here.

gopik avatar Oct 20 '21 07:10 gopik