incubator-uniffle
incubator-uniffle copied to clipboard
[#2181]improvement(server): Replace ShuffleBlockInfo with ShufflePartitionedBlock in netty
What changes were proposed in this pull request?
On the server side of the netty mode, when the server receives the sendShuffleData message, it decodes it into an unnecessary ShuffleBlockInfo and then converts it to ShufflePartitionedBlock in handleSendShuffleDataRequest.
Why are the changes needed?
Fix: #2181
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Locally
To ensure upgrade compatibility, only the server-side code was modified.
Test Results
2 926 files +10 2 926 suites +10 6h 14m 27s ⏱️ + 5m 19s 1 088 tests ± 0 1 086 ✅ + 1 2 💤 ±0 0 ❌ ±0 13 630 runs +10 13 600 ✅ +11 30 💤 ±0 0 ❌ ±0
Results for commit 17d4112d. ± Comparison against base commit 6e18b3e9.
:recycle: This comment has been updated with latest results.
Is this a improvement?
Is this a improvement?
In our stress test environment, it will generates 80million blocks. Removing useless ShuffleBlockInfo will reduce the JVM GC pressure.
@maobaolong Could you help me review this?
@lwllvyb With pleasure!
Codecov Report
Attention: Patch coverage is 62.06897% with 33 lines in your changes missing coverage. Please review.
Project coverage is 53.26%. Comparing base (
78fe934) to head (1a6b65d). Report is 15 commits behind head on master.
Additional details and impacted files
@@ Coverage Diff @@
## master #2182 +/- ##
============================================
+ Coverage 52.86% 53.26% +0.39%
+ Complexity 3385 3221 -164
============================================
Files 517 481 -36
Lines 28208 26073 -2135
Branches 2633 2467 -166
============================================
- Hits 14913 13887 -1026
+ Misses 12332 11281 -1051
+ Partials 963 905 -58
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@maobaolong Fixed
Ping @jerqi
@jerqi Let me take a final test for this PR
Use arthas to check the SendShuffleDataRequestV1 take effect, and use our pressure test cluster to verify this can works.
[arthas@264]$ vmtool --action getInstances --className org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1 --express 'instances'
@SendShuffleDataRequestV1[][
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@14581e0a],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@71dfeba4],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@6d86b40a],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@2502d413],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@717e36f7],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@7ff261b7],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@270cbca7],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@149d5619],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@145413f3],
@SendShuffleDataRequestV1[org.apache.uniffle.common.netty.protocol.SendShuffleDataRequestV1@78c73717],
Will merge this tomorrow. Thanks for your effort @lwllvyb and the review from @jerqi