tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

Published 20 hours ago • verified publisher icon pingcap

sink(ticdc): limit the number of transactions cached in a mysql worker (#10892)

Open ti-chi-bot opened this issue 10 months ago • 6 comments

This is an automated cherry-pick of #10892

What problem does this PR solve?

Issue Number: close #10896

What is changed and how it works?

  1. Remove the Add method of Txn.Worker to reduce interdependencies between modules.
  2. Use the method of conflictDetector to receive event in Txn.Worker.
  3. Implement causality.worker that support different batch control policies.

Check List

Tests

In the workload skew case, the maximum performance improvement after optimization is about 300% ~ 1000%.

  • Manual test (add detailed scripts or steps below)
  • Workload: sysbench random update
    1. Before #10376 image
    2. #10376 image
    3. With this pr image

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

`Fix the issue that workload could not be evenly distributed among different mysql sink workers in some scenarios`.

ti-chi-bot avatar Apr 22 '24 07:04 ti-chi-bot

/retest

CharlesCheung96 avatar Apr 28 '24 02:04 CharlesCheung96

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CharlesCheung96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • ~~OWNERS~~ [CharlesCheung96]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot[bot] avatar Apr 28 '24 02:04 ti-chi-bot[bot]

/retest

CharlesCheung96 avatar Apr 29 '24 03:04 CharlesCheung96

/retest

CharlesCheung96 avatar Apr 30 '24 09:04 CharlesCheung96

/retest

CharlesCheung96 avatar Apr 30 '24 10:04 CharlesCheung96

@ti-chi-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cdc-integration-mysql-test c68c96e018b0c98276c7b4965fab7f07cdf78c5d link true /test cdc-integration-mysql-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ti-chi-bot[bot] avatar Apr 30 '24 11:04 ti-chi-bot[bot]

/retest

CharlesCheung96 avatar May 20 '24 09:05 CharlesCheung96