spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-44953][CORE] Log a warning when shuffle tracking is enabled along side another DA supported mechanism

Open zwangsheng opened this issue 1 year ago • 4 comments

What changes were proposed in this pull request?

Log a warning when shuffle tracking is enabled along side another DA supported mechanism

Why are the changes needed?

Some users enable both shuffle tracking and another mechanism (like migration) and then are confused when their jobs don't scale down.

https://issues.apache.org/jira/browse/SPARK-44953

Does this PR introduce any user-facing change?

Yes, user can find the warning log when enabled both shuffle tracking and another DA supported mechanism(shuffle decommission).

How was this patch tested?

No

Was this patch authored or co-authored using generative AI tooling?

NO

zwangsheng avatar Mar 11 '24 03:03 zwangsheng

Hi @holdenk, could you please take a review this?

zwangsheng avatar Mar 11 '24 03:03 zwangsheng

LGTM

holdenk avatar Apr 25 '24 23:04 holdenk

It's been awhile (that's completely my fault) can you rebase and push and assuming CI still passes we can merge.

holdenk avatar Apr 25 '24 23:04 holdenk

Thanks for your review @holdenk, rebased this PR.

zwangsheng avatar Apr 26 '24 02:04 zwangsheng

Merged to current branch

holdenk avatar May 13 '24 20:05 holdenk