datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

Make fetch shuffle partition data in parallel

Open yahoNanJing opened this issue 3 years ago • 2 comments

Which issue does this PR close?

Closes #208.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

yahoNanJing avatar Sep 21 '22 04:09 yahoNanJing

Hi @andygrove, @thinkharderdev, @avantgardnerio, could you help review this PR?

yahoNanJing avatar Sep 21 '22 04:09 yahoNanJing

@yahoNanJing Could you please explain a little about the error handling case ? For example, if one of the parallel fetch failed, what happened to the others ? And what happened if there are multiple parallel fetch failures, will all the error info correctly propagated to the stream consumer ?

mingmwang avatar Sep 22 '22 06:09 mingmwang

@mingmwang Just added one commit to abort fast when error occurs.

yahoNanJing avatar Sep 27 '22 11:09 yahoNanJing