cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

kv/kvserver: TestStoreRangeSplitAndMergeWithGlobalReads failed

Open cockroach-teamcity opened this issue 1 year ago • 13 comments

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed with artifacts on master @ cc6ca026319024800395293b0fb18f05dd8eb50e:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/3c632028e5b231a582105881dfd5d4b5/logTestStoreRangeSplitAndMergeWithGlobalReads54447473
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.05s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-36100

cockroach-teamcity avatar Feb 15 '24 00:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed on master @ 617bf347978dcc0d711399b1a76402d7f88de958:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads331702955
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads331702955
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (52.99s)

Parameters:

  • attempt=1
  • run=9
  • shard=11
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 16 '24 07:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed on master @ 0b7ae19e2b94b851ed8812914f57032aab699811:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads2767376744
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads2767376744
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.01s)

Parameters:

  • attempt=1
  • run=12
  • shard=11
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 17 '24 07:02 cockroach-teamcity

Reproduces eagerly on master @ b8cca1ba:

dev test --stress --filter=TestStoreRangeSplitAndMergeWithGlobalReads pkg/kv/kvserver

Trying to bisect.

pav-kv avatar Feb 19 '24 15:02 pav-kv

This flake is not recent, e.g. the same failure occurs on 2c3b07ef from Jan 9.

pav-kv avatar Feb 19 '24 16:02 pav-kv

It looks like we're waiting for a metric counter to be equal to 1, but it jumps to 2, so we wait indefinitely. Not sure if it's legit (and we should just make the condition laxer) or a bug. Maybe we should wait for a more robust condition.

https://github.com/cockroachdb/cockroach/blob/a36097be277adef635f55d317579ca79b450bfef/pkg/kv/kvserver/client_split_test.go#L3688-L3699

@andrewbaptist @nvanbenschoten can you take a look since you last modified this test?

pav-kv avatar Feb 19 '24 16:02 pav-kv

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed on master @ a36097be277adef635f55d317579ca79b450bfef:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads3287400935
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads3287400935
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.01s)

Parameters:

  • attempt=1
  • run=23
  • shard=11
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 20 '24 07:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed with artifacts on master @ a78e1972a82f5b1bbb50d715aff46f7b668036fe:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/3c632028e5b231a582105881dfd5d4b5/logTestStoreRangeSplitAndMergeWithGlobalReads549076652
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/3c632028e5b231a582105881dfd5d4b5/logTestStoreRangeSplitAndMergeWithGlobalReads549076652
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.06s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 20 '24 09:02 cockroach-teamcity

@nvanbenschoten I can take a look at this. It definitely appears this is a test only change, and I didn't fully understand what was going on with this metric. The test is uglier with this change: https://github.com/cockroachdb/cockroach/commit/124aaa3b00349f4240c3a078301686c2d8883eda#diff-fb879e37911d655620c725817f9e37ffe56b226570268a3d31fabffd568a6c12L3678 since it now waits on a testutils.SucceedsSoon. A better way to structure this would have only the one range have GlobalReads set.

andrewbaptist avatar Feb 20 '24 20:02 andrewbaptist

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed on master @ e50b0ec4d3a53f81e26f3776ae3f3be55d435a9a:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads4231620982
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads4231620982
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.00s)

Parameters:

  • attempt=1
  • run=15
  • shard=11
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 21 '24 08:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed with artifacts on master @ 31acb7a07a4e6e1e96ceb8533cfa042ea80514a8:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/93fbc720f61b7e85f714b9750f754229/logTestStoreRangeSplitAndMergeWithGlobalReads180609409
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/93fbc720f61b7e85f714b9750f754229/logTestStoreRangeSplitAndMergeWithGlobalReads180609409
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.01s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 21 '24 12:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed with artifacts on master @ ee3168ac3e0286a63dd49ab8b9f14b036ad23bde:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/93fbc720f61b7e85f714b9750f754229/logTestStoreRangeSplitAndMergeWithGlobalReads1817694607
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/93fbc720f61b7e85f714b9750f754229/logTestStoreRangeSplitAndMergeWithGlobalReads1817694607
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (53.37s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 22 '24 01:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed on master @ 04f0416d526a43741d22fd03966758dcccdeb79f:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads3546424585
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads3546424585
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (52.99s)

Parameters:

  • attempt=1
  • run=19
  • shard=12
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 22 '24 12:02 cockroach-teamcity

kv/kvserver.TestStoreRangeSplitAndMergeWithGlobalReads failed on master @ 6d65201b9b603e0b3fcf1d509ec23edfdd68de45:

=== RUN   TestStoreRangeSplitAndMergeWithGlobalReads
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads3385134434
    test_log_scope.go:81: use -show-logs to present logs inline
    client_split_test.go:3691: condition failed to evaluate within 45s: from client_split_test.go:3693: commit wait count is 2
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStoreRangeSplitAndMergeWithGlobalReads3385134434
--- FAIL: TestStoreRangeSplitAndMergeWithGlobalReads (52.98s)

Parameters:

  • attempt=1
  • run=24
  • shard=12
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 23 '24 08:02 cockroach-teamcity

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.

github-actions[bot] avatar Apr 15 '24 10:04 github-actions[bot]