datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

Make scheduler prefer assign the task with same jobId to same executor

Open Ted-Jiang opened this issue 3 years ago • 0 comments

Is your feature request related to a problem or challenge? Please describe what you are trying to do. In current code, each poll work request will scan all waiting task in the same name space.https://github.com/apache/arrow-datafusion/blob/d7ae8c2631b5ea86ac2328530f0a4745daaf0bda/ballista/rust/scheduler/src/state/mod.rs#L283

plan1 -> shuffle wirte -> plan2

Seems p1 and p2 run on the same executor may avoid sending intermediate results. Describe the solution you'd like

  1. executor request job with jobId
  2. scheduler send same job task back

Describe alternatives you've considered

Ted-Jiang avatar Nov 19 '21 09:11 Ted-Jiang