roryqi

It's dangerous to delete the failed data of the stage when we retry. It's hard to reach the condition to delete the data. We should rely on the data skip...

> > It's dangerous to delete the failed data of the stage when we retry. It's hard to reach the condition to delete the data. > > Could you describe...

@EnricoMi If we have the retry of stage, the taskId may not unique. Because we don't have stage attemptId to differ task 1 attempt 0 in the stage attempt 0...

> > > > It's dangerous to delete the failed data of the stage when we retry. It's hard to reach the condition to delete the data. > > >...

> @EnricoMi If we have the retry of stage, the taskId may not unique. Because we don't have stage attemptId to differ task 1 attempt 0 in the stage attempt...

> Could you help review this? @EnricoMi @jerqi spark2 change will be finished after this PR is OK for you Several questions: 1. How to reject the legacy requests? 2....

> > > Spark client can easily come up with a per-stage-attempt shuffle id and feed that to the shuffle server. That should not require any server-side refactoring. > >...

> > Spark may compute partial tasks in a new attempt. > > You are saying a stage can be computed partially, let's say the first task and (if the...

> > If we make the unique shuffleIdWithAttemptNo generated or converted in server side > > I presume the server side does not know about the stage attempt number, so...

> > > > > If we make the unique shuffleIdWithAttemptNo generated or converted in server side > > > > > > > > > > > > I...