datafusion-ballista
                                
                                 datafusion-ballista copied to clipboard
                                
                                    datafusion-ballista copied to clipboard
                            
                            
                            
                        Make scheduler prefer assign the task with same jobId to same executor
Is your feature request related to a problem or challenge? Please describe what you are trying to do. In current code, each poll work request will scan all waiting task in the same name space.https://github.com/apache/arrow-datafusion/blob/d7ae8c2631b5ea86ac2328530f0a4745daaf0bda/ballista/rust/scheduler/src/state/mod.rs#L283
plan1 -> shuffle wirte -> plan2
Seems p1 and p2 run on the same executor may avoid sending intermediate results. Describe the solution you'd like
- executor request job with jobId
- scheduler send same job task back
Describe alternatives you've considered