incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

[Bug] Duplicated shaded-grpc jar cause task fail

Open lifeSo opened this issue 11 months ago • 3 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

Describe the bug

There is shaded-grpc jar in client-tez, When there is also shaded-grpc jar in user UDF, then, there will be sometimes task fail.

Uploading image.png…

Affects Version(s)

0.7.0

Uniffle Server Log Output

No response

Uniffle Engine Log Output

###|||2024-02-23 11:07:41.996|||INFO|||-|||-|||main|||-|||RssDAGAppMaster--->Registering coordinators 172.19.193.73:19999,172.19.193.182:19999
###|||2024-02-23 11:07:41.996|||INFO|||-|||-|||main|||-|||CoordinatorClientFactory--->Start to create coordinator clients from 172.19.193.73:19999,172.19.193.182:19999
###|||2024-02-23 11:07:42.013|||ERROR|||-|||-|||main|||-|||RssDAGAppMaster--->Error starting RssDAGAppMaster
java.lang.NoSuchFieldError: EPOLL_DOMAIN_CLIENT_CHANNEL_TYPE
        at io.grpc.netty.shaded.io.grpc.netty.UdsNettyChannelProvider.isAvailable(UdsNettyChannelProvider.java:34)
        at io.grpc.ManagedChannelRegistry$ManagedChannelPriorityAccessor.isAvailable(ManagedChannelRegistry.java:172)
        at io.grpc.ManagedChannelRegistry$ManagedChannelPriorityAccessor.isAvailable(ManagedChannelRegistry.java:168)
        at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:68)
        at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:96)
        at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:41)
        at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
        at org.apache.uniffle.client.impl.grpc.GrpcClient.<init>(GrpcClient.java:43)
        at org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient.<init>(CoordinatorGrpcClient.java:91)
        at org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient.<init>(CoordinatorGrpcClient.java:87)
        at org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient.<init>(CoordinatorGrpcClient.java:83)
        at org.apache.uniffle.client.factory.CoordinatorClientFactory.createCoordinatorClient(CoordinatorClientFactory.java:43)
        at org.apache.uniffle.client.factory.CoordinatorClientFactory.createCoordinatorClient(CoordinatorClientFactory.java:69)
        at org.apache.uniffle.client.impl.ShuffleWriteClientImpl.registerCoordinators(ShuffleWriteClientImpl.java:431)
        at org.apache.tez.dag.app.RssDAGAppMaster.initAndStartRSSClient(RssDAGAppMaster.java:144)
        at org.apache.tez.dag.app.RssDAGAppMaster.serviceInit(RssDAGAppMaster.java:107)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
        at org.apache.tez.dag.app.RssDAGAppMaster.main(RssDAGAppMaster.java:348)

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

lifeSo avatar Feb 28 '24 05:02 lifeSo

In tez-client: image

There is no "EPOLL_DOMAIN_CLIENT_CHANNEL_TYPE" field in user shaded-grpc jar: image

lifeSo avatar Feb 28 '24 05:02 lifeSo

Could you help fix this? @lifeSo

zuston avatar Mar 14 '24 02:03 zuston

Could you help fix this? @lifeSo

OK, I could try, remove or rename the shaded jar name to other, to not conflict user's jar.

lifeSo avatar Mar 18 '24 08:03 lifeSo