tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

changefeed lag spikes for about 10 min every 1 hours

Open fubinzh opened this issue 2 years ago • 5 comments

What did you do?

  1. Start changefeed
  2. Run workload
  3. Check changefeed lag

What did you expect to see?

Changefeed lag should be stable

What did you see instead?

changefeed lag spikes for about 10 min every 1 hours. The latency spike seems have the same pace as PD region scheduler.

img_v2_2fbc5394-0bbd-4300-ad67-eed18205d7dg

586e5f5a-8629-497e-b018-66c0254cc699

Versions of the cluster

Master

fubinzh avatar Sep 20 '22 09:09 fubinzh

@fubinzh: The label(s) enhancement cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, affects-6.3, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, may-affects-6.3, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot avatar Sep 20 '22 09:09 ti-chi-bot

@fubinzh: The label(s) type/enhancement cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, affects-6.3, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, may-affects-6.3, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label type/enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot avatar Sep 20 '22 09:09 ti-chi-bot

/type enhancement

fubinzh avatar Sep 20 '22 09:09 fubinzh

@fubinzh: The label(s) type/enhancement cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, affects-6.3, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, may-affects-6.3, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label type/enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot avatar Sep 20 '22 09:09 ti-chi-bot

@fubinzh: The label(s) enhancement cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, affects-6.3, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, may-affects-6.3, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, needs-cherry-pick-release-6.3, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot avatar Sep 20 '22 09:09 ti-chi-bot