incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

[Improvement] ShuffleBlockInfo.getSize can be optimized when enabling Netty using off-heap memory

Open rickyma opened this issue 1 year ago • 2 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

As the title says, we use off-heap memory when enabling Netty, so there is no need to add the metadata size of fields.

How should we improve?

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

rickyma avatar Jul 16 '24 03:07 rickyma

Hi @rickyma

Based on my understanding of the description, we need to modify the following part from sbi.getSize() to sbi.getLength(). Please help to confirm my understanding, thank you. 😄

https://github.com/apache/incubator-uniffle/blob/3ea5c2c780b1cb8cf01e63ecd82b5c2410c2ac49/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java#L514

https://github.com/apache/incubator-uniffle/blob/3ea5c2c780b1cb8cf01e63ecd82b5c2410c2ac49/internal-client/src/main/java/org/apache/uniffle/client/impl/grpc/ShuffleServerGrpcClient.java#L555

https://github.com/apache/incubator-uniffle/blob/3ea5c2c780b1cb8cf01e63ecd82b5c2410c2ac49/internal-client/src/main/java/org/apache/uniffle/client/impl/grpc/ShuffleServerGrpcNettyClient.java#L145

cchung100m avatar Jan 08 '25 15:01 cchung100m

It's been a long time. I don't know if the code has already been optimized by @maobaolong in some other way.

rickyma avatar Jan 09 '25 06:01 rickyma