tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

One of TiKV min resolved lag is 3 hours during sysbench prepare

Open Tammyxia opened this issue 3 years ago • 0 comments

What did you do?

  • Test-infra case, 1 changefeed with sink MySQL, base image is alpine.
  • workload: sysbench oltp_write_only prepare --tables=128 --table-size=10000000 --time=86400 --debug=true --threads=128 (UTC 2022-05-18T07:20:20 to 2022-05-19T07:30:20) (Beijing 2022-05-18T15:20:20 to 2022-05-19T15:30:20)
  • sysbench oltp_write_only run --tables=128 --table-size=10000000 --time=86400 --debug=true --threads=128 (UTC 2022-05-19T07:30:20 to 2022/05/20 07:30:23) (Beijing 2022-05-19T15:30:20 to 2022/05/20 15:30:23)

What did you expect to see?

No response

What did you see instead?

  • TiKV-1 min resolved lag is 3 hours during sysbench prepare, other TiKV min resolved lag is several seconds. image

  • Only TiCDC-1 checkpoint lag is 3 hours, while other TiCDC checkpoint lag is < 10 min. image

  • TiDB: image

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)
v6.1.0-nightly  2022-05-16

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)
Release Version:   6.1.0-alpha
Edition:           Community
Git Commit Hash:   748f8a95411fc95e05ebc039d233cdfa04e4acb9
Git Commit Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time:    2022-05-16 07:30:24
Rust Version:      rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Enable Features:   jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

TiCDC version (execute cdc version):

(paste TiCDC version here)
Release Version: v6.1.0-nightly
Git Commit Hash: 4a992703a8bd67d173ff4a9d3d9ad8fdda189432
Git Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time: 2022-05-16 07:29:46
Go Version: go version go1.18 linux/amd64
Failpoint Build: false

Tammyxia avatar May 20 '22 08:05 Tammyxia