Junfan Zhang comments

Results 434 comments of


                                            Junfan Zhang

[Bug] fix(spark): Triggering Stage retry requires reassigning the shuffle server in the retry Stage.

> Modify relevant code. Thanks for your effort, let's reopen this feature together.

[#2061]fix(server): Fix netty memory leak when removeBuffer and cacheShuffle…

BTW, If I missed somthing, feel free to ping me @lwllvyb or by wechat

[#2503] fix(spark): Skip reassignment number check on partition split

Gentle ping @yl09099 @jerqi .

fix: Add option with "--privileged=true" to run by docker

> Thanks @zuston for pushing the PR. > > However, I am not sure that this is a good idea as privileged mode enables roiot on host system and is...

[#2508] feat(spark3): Record failed tasks on any shuffle write/failure into event logs

Update @jerqi

[#2494] feat(spark): Overlapping compression to avoid blocking shuffle writing

Maybe this is not a best strategy to reduce the compression cost, but this is acceptable. Actually the biggest cost of compression is the frequent memory allocation, if we could...

[#2494] feat(spark): Overlapping compression to avoid blocking shuffle writing

cc @jerqi

[#2494] feat(spark): Overlapping compression to avoid blocking shuffle writing

> Could you give some data for this improvement? Still in evaluating and I will explore the compression improvements

[#2494] feat(spark): Overlapping compression to avoid blocking shuffle writing

> Could you give some data for this improvement? Updated in the description.

[#1938] improvement: Use ShuffleSegment to replace FileBasedShuffleSegment and BufferSegment

Sorry for the late reply. @wankunde Thanks for proposing this. Actually the `ShuffleSegment` has been introduce to be as the an abstract class. And I think the initial design is...