datafusion-ballista
datafusion-ballista copied to clipboard
feat: consistent hash scheduling implemented as `DistributionPolicy`
Which issue does this PR close?
Closes #.
Rationale for this change
pluggable DistributionPolicy provides a way to extend scheduler task binding policies.
with consistent hash task binding policy being quite use case specific implementing it as DistributionPolicy would make a sense. This would reduce api surface we need to maintain.
follow up needed:
- improved tests
- example how to use it
- documentation
What changes are included in this PR?
- custom
DistributionPolicyfor consistent hash scheduling - moved to its own submodule
- policy has been removed from out of the box policy, users should configure it programmatically
Are there any user-facing changes?
- consistent hash option removed from scheduler configuration