Junfan Zhang comments

Results 434 comments of


                                            Junfan Zhang

[#1796] fix(spark): Implicitly unregister map output on fetch failure

Could you help review this? @advancedxy

[#1796] fix(spark): Implicitly unregister map output on fetch failure

> Thanks for working on this. I did a quick overview about this change, I think it's quite large to review. It would best to keep this pr open and...

[#1796] fix(spark): Implicitly unregister map output on fetch failure

Good to know this. Thanks for your determined reply. @advancedxy

[#1796] fix(spark): Implicitly unregister map output on fetch failure

> However, I think this PR should be split into two PRs at least: handle fetch failure and handle write failures. I think the write failure is different from fetch...

[#1796] fix(spark): Implicitly unregister map output on fetch failure

After digging into this feature, I found there are many bugs and improvement need to be done. So I have to split them into small patch to fix. And this...

[Bug] Remove handlers when purging app shuffleIds

> Will this issue lead to some more serious problems? Noop. Just remains some unnecessary data, but this is critial for stage retry

[#2148] improvement(server): Rename ShuffleBufferWithLinkedList to DefaultShuffleBuffer

cc @maobaolong

Rework stage retry

> > The stage retry whole design is based on the Spark's fetchFailedException , once spark scheduler accepts this exception thrown from task, it will kill all the running tasks...

Rework stage retry

To ensure the data consistency, I will support the original unique task attempt id in blockId layout for the stage retry to filter out the previous stage data. PTAL @EnricoMi

Rework stage retry

I don't find out any concept that the limit operator will get the stage attempt 0 + stage attempt 1 data to be as the normal data pipeline. Could you...