incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

[#2181]improvement(server): Replace ShuffleBlockInfo with ShufflePartitionedBlock in netty

Open lwllvyb opened this issue 1 year ago • 9 comments

What changes were proposed in this pull request?

On the server side of the netty mode, when the server receives the sendShuffleData message, it decodes it into an unnecessary ShuffleBlockInfo and then converts it to ShufflePartitionedBlock in handleSendShuffleDataRequest.

Why are the changes needed?

Fix: #2181

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Locally

lwllvyb avatar Oct 15 '24 06:10 lwllvyb

To ensure upgrade compatibility, only the server-side code was modified.

lwllvyb avatar Oct 15 '24 06:10 lwllvyb

Test Results

 2 926 files  +10   2 926 suites  +10   6h 14m 27s ⏱️ + 5m 19s  1 088 tests ± 0   1 086 ✅ + 1   2 💤 ±0  0 ❌ ±0  13 630 runs  +10  13 600 ✅ +11  30 💤 ±0  0 ❌ ±0 

Results for commit 17d4112d. ± Comparison against base commit 6e18b3e9.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Oct 15 '24 06:10 github-actions[bot]

Is this a improvement?

jerqi avatar Oct 18 '24 06:10 jerqi

Is this a improvement?

In our stress test environment, it will generates 80million blocks. Removing useless ShuffleBlockInfo will reduce the JVM GC pressure.

lwllvyb avatar Oct 18 '24 07:10 lwllvyb

@maobaolong Could you help me review this?

jerqi avatar Oct 18 '24 07:10 jerqi

@lwllvyb With pleasure!

maobaolong avatar Oct 18 '24 07:10 maobaolong

Codecov Report

Attention: Patch coverage is 62.06897% with 33 lines in your changes missing coverage. Please review.

Project coverage is 53.26%. Comparing base (78fe934) to head (1a6b65d). Report is 15 commits behind head on master.

Files with missing lines Patch % Lines
...ommon/netty/protocol/SendShuffleDataRequestV1.java 69.81% 16 Missing :warning:
.../common/netty/protocol/SendShuffleDataRequest.java 0.00% 9 Missing :warning:
...niffle/server/netty/ShuffleServerNettyHandler.java 0.00% 8 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2182      +/-   ##
============================================
+ Coverage     52.86%   53.26%   +0.39%     
+ Complexity     3385     3221     -164     
============================================
  Files           517      481      -36     
  Lines         28208    26073    -2135     
  Branches       2633     2467     -166     
============================================
- Hits          14913    13887    -1026     
+ Misses        12332    11281    -1051     
+ Partials        963      905      -58     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Oct 18 '24 11:10 codecov-commenter

@maobaolong Fixed

lwllvyb avatar Oct 18 '24 11:10 lwllvyb

Ping @jerqi

lwllvyb avatar Oct 22 '24 07:10 lwllvyb

@jerqi Let me take a final test for this PR

maobaolong avatar Oct 30 '24 12:10 maobaolong

Use arthas to check the SendShuffleDataRequestV1 take effect, and use our pressure test cluster to verify this can works.

[arthas@264]$ vmtool  --action getInstances --className org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1 --express 'instances'
@SendShuffleDataRequestV1[][
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@14581e0a],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@71dfeba4],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@6d86b40a],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@2502d413],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@717e36f7],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@7ff261b7],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@270cbca7],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@149d5619],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@145413f3],
    @SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@78c73717],

Will merge this tomorrow. Thanks for your effort @lwllvyb and the review from @jerqi

maobaolong avatar Oct 30 '24 15:10 maobaolong