Mridul M

Merged to master. Thanks for working on this @wankunde ! Thanks for the review @otterc :-)

Parent directory would not be typically removed unless we are shutting down the container (for ex) - and in those cases usually, we should not be recreating the parent dirs....

Can you update the description with what is the behavior we are actually observing ? The details in jira and PR description does not detail what the issue is, just...

If a result stage does not have pending partitions, it does not need to be aborted - since there are no partitions to be computed. If a result stage has...

+CC @ueshin, as you reviewed the surrounding code before. Also, CC @HyukjinKwon.

+CC @otterc, @Ngone51

+CC @akpatnam25, @otterc - since you were planning to work on this already.

We should decouple current implementation details when making protocol changes, and make it extensible for future evolution. In this case though, it is much more straightforward - there is an...

> The push-based shuffle service will auto clean up the old shuffle merge data Consider the case I mentioned above - stage retry for an `INDETERMINATE` stage. We cleanup previous...

I am unsure of this test - will rely on @tgravescs to comment better.