Junfan Zhang comments

Results 434 comments of


                                            Junfan Zhang

[#1239] Remote merge on the shuffle server side.

> > > For this particular feature, I think MR/Tez would benefit from that substantially(to be frankly, I had developed a similar shuffle system back ~6-7 years ago). The spark...

[#1239] Remote merge on the shuffle server side.

Thanks for your work. @zhengchenyu After reviewing the design doc and concrete code briefly, I have some question about this feature. ### Spark client 1. What's the diference of RecordBlob...

[#1239] Remote merge on the shuffle server side.

Firstly I think this is meaningful for tez/mr. For spark, the sorted shuffle is still meaningful, like the facebook did similar things in their cosco (facebook internal remote shuffle service),...

[#1699] improvement: Upgrade from commons-collections:commons-collections:3.2.2 to org.apache.commons:commons-collections:4.5.0-M1

Is this common update for other opensource projects?

[#1608][part-4] feat(server)(spark3): activate partition reassign when server is inactive

> This function is very useful, but I'm a little worried. when all apps in the cluster are enabled, if server a1 is inactive, it will be immediately assigned to...

[#1608][part-4] feat(server)(spark3): activate partition reassign when server is inactive

PTAL @xumanbu @dingshun3016 @rickyma @jerqi

[#1608][part-4] feat(server)(spark3): activate partition reassign when server is inactive

All done @jerqi

[#1608][part-4] feat(server)(spark3): activate partition reassign when server is inactive

> In general, I want to discuss with some design ideas. What's the scope of this pull request to handle? Unhealthy node? Decomission node? You create a new concept inactive...

Support client partition data reassign

> > I found some bugs and design problems > > What kind of bugs did you find? What will the bug cause? Data loss, or ... ? Can you...

[#1579][part-1] fix(spark): Adjust reassigned time to ensure that all previous data is cleared for stage retry

cc @dingshun3016 @yl09099 PTAL