datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

feat: consistent hash scheduling implemented as `DistributionPolicy`

Open milenkovicm opened this issue 10 months ago • 0 comments

Which issue does this PR close?

Closes #.

Rationale for this change

pluggable DistributionPolicy provides a way to extend scheduler task binding policies. with consistent hash task binding policy being quite use case specific implementing it as DistributionPolicy would make a sense. This would reduce api surface we need to maintain.

follow up needed:

  • improved tests
  • example how to use it
  • documentation

What changes are included in this PR?

  • custom DistributionPolicy for consistent hash scheduling
  • moved to its own submodule
  • policy has been removed from out of the box policy, users should configure it programmatically

Are there any user-facing changes?

  • consistent hash option removed from scheduler configuration

milenkovicm avatar Jun 15 '25 09:06 milenkovicm