Mridul M

We ask users to copy celeborn client side jar to spark jars directory - that should ensure this problem does not occur, right ? So this would apply only when...

@SteNicholas That makes sense, there are definitely scenarios where we cannot update the spark distribution with additional jars. The reason why I brought that up is, currently the warning will...

Hi @SteNicholas, which modes did you test this in ? I did a quick check against local cluster, spark standalone and local yarn - and keep getting this and variants...

~Let me recheck that - I think the celeborn diff did not apply in my local build~ Works fine with celeborn jar in spark jar directory, not with --jars for...

Hi @SteNicholas , I was able to reproduce this behavior when celeborn jar is there in both spark platform jars and is specified by users via `--jars`. Does this match...

Before 4.0, Apache Spark does not support shuffle manager from user jars - so I am not sure how this is working for you ... unless the fix was cherry...

A general query on this - is there any benefit to not supporting auth going forward given authn support has completed ? (except for backward compatibility reasons for older applications)...

+CC @xkrogen

I will try to get to this later this week, do let me know if you are still working on it/have pending comments to address. Thanks

+CC @otterc, can you take a look at this PR ?