tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

mounter(ticdc): optimize mounter cpu usage by reduce memory allocation on map and slice

Open 3AceShowHand opened this issue 9 months ago • 8 comments

What problem does this PR solve?

Issue Number: close #11210

What is changed and how it works?

  • use sync.Pool to reduce the map allocation for the datum
  • remove columnInfo slice from the datums2Column method, to reduce memory allocation.

Check List

Tests

  • Unit test
  • Integration test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`None`

3AceShowHand avatar May 13 '24 10:05 3AceShowHand

Codecov Report

Attention: Patch coverage is 92.15686% with 4 lines in your changes missing coverage. Please review.

Project coverage is 57.6113%. Comparing base (7fbae79) to head (57b01d9). Report is 18 commits behind head on master.

:exclamation: Current head 57b01d9 differs from pull request most recent head f59703a

Please upload reports for the commit f59703a to get more accurate results.

Additional details and impacted files
Components Coverage Δ
cdc 61.4317% <92.1568%> (-0.0518%) :arrow_down:
dm 51.1638% <ø> (-0.0424%) :arrow_down:
engine 63.4020% <ø> (+0.0494%) :arrow_up:
Flag Coverage Δ
unit 57.6113% <92.1568%> (-0.0363%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11081        +/-   ##
================================================
- Coverage   57.6475%   57.6113%   -0.0363%     
================================================
  Files           850        850                
  Lines        126040     126055        +15     
================================================
- Hits          72659      72622        -37     
- Misses        47985      48026        +41     
- Partials       5396       5407        +11     

codecov[bot] avatar May 30 '24 10:05 codecov[bot]

Please add the test result.

CharlesCheung96 avatar May 31 '24 05:05 CharlesCheung96

Please add the test result.

The metrics is lost, since I use continous profiling which delete metrics data each 3 hours.

On the master branch, we can see obvious makemap called inside the decodedRowV2 method, and makeslice around 2.0% CPU usage inside the datum2Columns method.

In this PR, the makemap cannot be seen, and makeslice reduced to around 1.1%.

3AceShowHand avatar May 31 '24 06:05 3AceShowHand

[LGTM Timeline notifier]

Timeline:

  • 2024-05-31 08:04:22.765480925 +0000 UTC m=+3022816.522616496: :ballot_box_with_check: agreed by CharlesCheung96.

ti-chi-bot[bot] avatar May 31 '24 08:05 ti-chi-bot[bot]

Could you please provide a benchmark test result? It would be really helpful.

asddongmen avatar Jun 03 '24 09:06 asddongmen

The metrics is lost, since I use continous profiling which delete metrics data each 3 hours.

On the master branch, we can see obvious makemap called inside the decodedRowV2 method, and makeslice around 2.0% CPU usage inside the datum2Columns method.

In this PR, the makemap cannot be seen, and makeslice reduced to around 1.1%.

The metrics is lost, since I use continous profiling which delete metrics data each 3 hours.

On the master branch, we can see obvious makemap called inside the decodedRowV2 method, and makeslice around 2.0% CPU usage inside the datum2Columns method.

In this PR, the makemap cannot be seen, and makeslice reduced to around 1.1%.

3AceShowHand avatar Jun 03 '24 10:06 3AceShowHand

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, CharlesCheung96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • ~~OWNERS~~ [CharlesCheung96,asddongmen]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot[bot] avatar Jun 17 '24 02:06 ti-chi-bot[bot]

@3AceShowHand: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cdc-integration-pulsar-test f59703ae3c583e094370068a597537791092805f link unknown /test cdc-integration-pulsar-test
jenkins-ticdc/verify f59703ae3c583e094370068a597537791092805f link unknown /test verify
pull-cdc-integration-mysql-test f59703ae3c583e094370068a597537791092805f link unknown /test cdc-integration-mysql-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ti-chi-bot[bot] avatar Jun 17 '24 03:06 ti-chi-bot[bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ti-chi-bot[bot] avatar Jul 05 '24 08:07 ti-chi-bot[bot]