cockroach roachtest: kv/gracefuldraining failed

roachtest.kv/gracefuldraining failed with artifacts on master @ b6c13686495bbe9ad476b28033461ef7628e18a8:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:621
	            				main/pkg/cmd/roachtest/monitor.go:120
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1207.00 at time 2024-09-29 07:20:30.262333932 +0000 UTC m=+5522.211238862 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:604
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:621
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:120
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1207.00 at time 2024-09-29 07:20:30.262333932 +0000 UTC m=+5522.211238862 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:642).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

Jira issue: CRDB-42610

Sep 29 '24 07:09 cockroach-teamcity

The latency is reflected in the workload logs as well, 25s after restarting n6, after stopping n6 once.

# (07:20:28)
  151.0s        0          504.8          496.0      1.4      1.9      3.1     13.1 follower-read
  151.0s        0          489.8          493.3      1.5      1.9      2.4      6.6 read
  151.0s        0         1003.7          993.8      3.8      4.7      5.5     18.9 write
  152.0s        0          346.1          495.0      1.4      2.0      2.6      3.3 follower-read
  152.0s        0          306.1          492.1      1.5      2.1      2.6      3.9 read
  152.0s        0          653.2          991.5      3.8      5.0      5.5      7.1 write
  153.0s        0            7.0          491.8      1.4      1.9      1.9      1.9 follower-read
  153.0s        0           13.0          488.9   1342.2   1543.5   1610.6   1610.6 read
  153.0s        0           14.0          985.1      3.9      4.7      5.0      5.0 write

There's some correlated latency jumps for clock sync to n1 right as the test fails (n2,n3,n4,n5,n6):

Details

W240929 07:20:29.524490 239 2@rpc/clock_offset.go:291 â‹® [T1,Vsystem,n2,rnode=1,raddr=â€¹10.128.0.145:26257â€º,class=default,rpc] 733  latency jump (prev avg 0.63ms, current 934.24ms)
W240929 07:20:29.602120 518 2@rpc/clock_offset.go:291 â‹® [T1,Vsystem,n3,rnode=1,raddr=â€¹10.128.0.145:26257â€º,class=raft,rpc] 70  latency jump (prev avg 0.61ms, current 647.79ms)
W240929 07:20:29.623591 277 2@rpc/clock_offset.go:291 â‹® [T1,Vsystem,n4,rnode=1,raddr=â€¹10.128.0.145:26257â€º,class=system,rpc] 729  latency jump (prev avg 0.56ms, current 1131.96ms)
W240929 07:20:29.622413 477 2@rpc/clock_offset.go:291 â‹® [T1,Vsystem,n5,rnode=1,raddr=â€¹10.128.0.145:26257â€º,class=rangefeed,rpc] 712  latency jump (prev avg 0.59ms, current 916.45ms)
W240929 07:20:29.593994 714 2@rpc/clock_offset.go:291 â‹® [T1,Vsystem,n6,rnode=1,raddr=â€¹10.128.0.145:26257â€º,class=default,rpc] 86  latency jump (prev avg 0.75ms, current 748.68ms)

Which suggest n1 slowed down processing right at the moment throughput dropped. This is supported by raft slow logging and slow heartbeats:

I240929 07:20:29.523266 329 kv/kvserver/store_raft.go:697 â‹® [T1,Vsystem,n1,s1,r73/1:â€¹/Table/106/1/-7{94488â€¦-76224â€¦}â€º,raft] 1013  raft ready handling: 0.51s [append=0.00s, apply=0.00s, non-blocking-sync=0.00s, other=0.51s], wrote [append-batch=710 B, append-ent=456 B (4), ]; node might be overloaded
W240929 07:20:30.173956 354 kv/kvserver/liveness/liveness.go:758 â‹® [T1,Vsystem,n1,liveness-hb] 1027  slow heartbeat took 1.137388463s; err=<nil>

Sure enough, there's large spike in goroutine scheduling latency on n1 at this time:

Which correlates with the exec latency spike:

Sadly, there's no CPU profile for this period and I can't glean why there was a goroutine spike, aside from saying it may not be replica related based on the recorded CPU (but this may not capture the spike):

We need to capture a CPU profile during the spike, or a runtime trace.

Marking as a bug and assigning P-2 for now. Opened https://github.com/cockroachdb/cockroach/pull/131622 to profile CPU continuously.

Sep 30 '24 20:09 kvoli

roachtest.kv/gracefuldraining failed with artifacts on master @ ec2573dc6aaeefc226440bb2c5a7c94a63989868:

(cluster.go:2370).Start: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 02 '24 10:10 cockroach-teamcity

Ah looks like 0 wasn't valid for the profiling interval. Opened https://github.com/cockroachdb/cockroach/pull/131760. I'll try a run myself and see what happens as well.

Oct 02 '24 16:10 kvoli

I'm going to remove my assignment -- as this involvement was primarily triage related. If the test fails again with the CPU profiles available, I'll take a look.

Oct 03 '24 16:10 kvoli

roachtest.kv/gracefuldraining failed with artifacts on master @ f842c3b4b5adc040d411bd17d7d10005273fc1b6:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1511.00 at time 2024-10-05 11:08:14.955073169 +0000 UTC m=+19126.742523178 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1511.00 at time 2024-10-05 11:08:14.955073169 +0000 UTC m=+19126.742523178 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2478).Run: context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 05 '24 11:10 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on master @ dcce4cafa234525fc859d32745c11ed87890dc7b:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1584.00 at time 2024-10-06 07:29:48.665942679 +0000 UTC m=+6097.314197839 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1584.00 at time 2024-10-06 07:29:48.665942679 +0000 UTC m=+6097.314197839 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 06 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ dcce4cafa234525fc859d32745c11ed87890dc7b:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1533.00 at time 2024-10-07 11:13:24.886998512 +0000 UTC m=+19497.770468466 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1533.00 at time 2024-10-07 11:13:24.886998512 +0000 UTC m=+19497.770468466 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 07 '24 11:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 1f963ef16416e17ddb394509426b00a340700f9e:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1721.00 at time 2024-10-08 07:18:54.150536058 +0000 UTC m=+5407.602673293 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1721.00 at time 2024-10-08 07:18:54.150536058 +0000 UTC m=+5407.602673293 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 08 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 58c475d67e32b75284b4fe293bff82807c3d129d:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1767.00 at time 2024-10-09 07:26:13.239845816 +0000 UTC m=+5796.232932366 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1767.00 at time 2024-10-09 07:26:13.239845816 +0000 UTC m=+5796.232932366 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2415).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 09 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ fd4b1464dbd6e385c6e51af26fe294fd2023a259:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1624.00 at time 2024-10-10 07:15:50.410420869 +0000 UTC m=+5108.646975239 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1624.00 at time 2024-10-10 07:15:50.410420869 +0000 UTC m=+5108.646975239 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2478).Run: context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 10 '24 07:10 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1541.00 at time 2024-10-11 07:03:49.618791969 +0000 UTC m=+4444.354326762 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1541.00 at time 2024-10-11 07:03:49.618791969 +0000 UTC m=+4444.354326762 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 11 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1700
	Error:      	Received unexpected error:
	            	QPS of 1658.00 at time 2024-10-12 08:20:20.880215855 +0000 UTC m=+9276.525975137 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1700
	            	Wraps: (2) QPS of 1658.00 at time 2024-10-12 08:20:20.880215855 +0000 UTC m=+9276.525975137 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:644).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 12 '24 08:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1700
	Error:      	Received unexpected error:
	            	QPS of 1465.00 at time 2024-10-13 06:38:06.375622881 +0000 UTC m=+3157.909751622 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1700
	            	Wraps: (2) QPS of 1465.00 at time 2024-10-13 06:38:06.375622881 +0000 UTC m=+3157.909751622 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 13 '24 06:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1700
	Error:      	Received unexpected error:
	            	QPS of 1740.00 at time 2024-10-14 07:14:25.678945621 +0000 UTC m=+5335.355758511 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1700
	            	Wraps: (2) QPS of 1740.00 at time 2024-10-14 07:14:25.678945621 +0000 UTC m=+5335.355758511 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 14 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 49ca24cedb042579e9645c206640d59975805d12:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1700
	Error:      	Received unexpected error:
	            	QPS of 1640.00 at time 2024-10-15 07:29:26.063306377 +0000 UTC m=+6146.675607323 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1700
	            	Wraps: (2) QPS of 1640.00 at time 2024-10-15 07:29:26.063306377 +0000 UTC m=+6146.675607323 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:644).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 15 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 5be5b0b52ff79b98689b2282a8b25cf9eb50ec40:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1700
	Error:      	Received unexpected error:
	            	QPS of 1588.00 at time 2024-10-16 11:56:50.611487054 +0000 UTC m=+22217.580666349 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1700
	            	Wraps: (2) QPS of 1588.00 at time 2024-10-16 11:56:50.611487054 +0000 UTC m=+22217.580666349 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 16 '24 11:10 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on master @ 42f40f59cae3c0fd8842e194d6991c951ab4382f:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1798.00 at time 2024-10-17 07:44:54.541747312 +0000 UTC m=+6858.959234860 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1798.00 at time 2024-10-17 07:44:54.541747312 +0000 UTC m=+6858.959234860 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 17 '24 07:10 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on master @ 833dadd212fa4b12b1442ae8e00e85ee80a8cdce:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1655.00 at time 2024-10-18 06:51:49.189965712 +0000 UTC m=+3600.586334935 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1655.00 at time 2024-10-18 06:51:49.189965712 +0000 UTC m=+3600.586334935 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 18 '24 06:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1621.00 at time 2024-10-19 07:31:32.576624103 +0000 UTC m=+6157.515541651 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1621.00 at time 2024-10-19 07:31:32.576624103 +0000 UTC m=+6157.515541651 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 19 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1766.00 at time 2024-10-20 11:12:09.110148978 +0000 UTC m=+19368.523315909 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1766.00 at time 2024-10-20 11:12:09.110148978 +0000 UTC m=+19368.523315909 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:644).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv branch-release-24.3 release-blocker]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 20 '24 11:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1739.00 at time 2024-10-21 11:21:39.914379584 +0000 UTC m=+19857.849327835 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1739.00 at time 2024-10-21 11:21:39.914379584 +0000 UTC m=+19857.849327835 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv branch-release-24.3 release-blocker]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 21 '24 11:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 1e5b3c212b45419c960038718c48a5dd75a111a0:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1725.00 at time 2024-10-22 07:36:11.500681797 +0000 UTC m=+6271.757264008 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1725.00 at time 2024-10-22 07:36:11.500681797 +0000 UTC m=+6271.757264008 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:644).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 22 '24 07:10 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on master @ 787f2e3fe5f73b33fcd65485908cbb71e0991222:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1659.00 at time 2024-10-23 07:30:12.203359154 +0000 UTC m=+6042.573700178 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1659.00 at time 2024-10-23 07:30:12.203359154 +0000 UTC m=+6042.573700178 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 23 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 9c1d89e3adb8c6532459cc3e616288db06f966d9:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1794.00 at time 2024-10-24 11:13:40.019344736 +0000 UTC m=+19403.027759003 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1794.00 at time 2024-10-24 11:13:40.019344736 +0000 UTC m=+19403.027759003 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 24 '24 11:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 82b1fda15c4616713b278c447d24b0ab5416e511:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1638.00 at time 2024-10-25 07:21:37.869589264 +0000 UTC m=+5488.202415917 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1638.00 at time 2024-10-25 07:21:37.869589264 +0000 UTC m=+5488.202415917 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:644).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 25 '24 07:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 7495c0390f93478bc0f961285f41cd4ec85a6a76:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1763.00 at time 2024-10-26 11:31:41.28558219 +0000 UTC m=+20427.727846316 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1763.00 at time 2024-10-26 11:31:41.28558219 +0000 UTC m=+20427.727846316 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:695).drainWithIpTables: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 26 '24 11:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 1b5c41939197efb0fa50ded9795e2e83f5c1dd34:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            				main/pkg/cmd/roachtest/monitor.go:115
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1710.00 at time 2024-10-28 11:26:40.935068343 +0000 UTC m=+20191.419794430 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:606
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:623
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1710.00 at time 2024-10-28 11:26:40.935068343 +0000 UTC m=+20191.419794430 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:644).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 28 '24 11:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ 615110b2b10de95d6ea25602db5dd8fdd5dbe576:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:632
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 1747.00 at time 2024-10-29 06:49:00.164557962 +0000 UTC m=+3926.266283178 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:615
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:632
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 1747.00 at time 2024-10-29 06:49:00.164557962 +0000 UTC m=+3926.266283178 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(kv.go:653).func1: context cancelled while waiting
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 29 '24 06:10 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/gracefuldraining failed with artifacts on master @ ffe1b9fed2b7ee3b8d53d6d943038c358b8eb5a6:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:632
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 553.00 at time 2024-10-30 09:59:43.641721778 +0000 UTC m=+15790.761535532 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:615
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:632
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 553.00 at time 2024-10-30 09:59:43.641721778 +0000 UTC m=+15790.761535532 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2386).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 30 '24 10:10 cockroach-teamcity

roachtest.kv/gracefuldraining failed with artifacts on master @ becbd0fcdfa2e37a6ff23b33af70f2f91eca0790:

(assertions.go:363).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/kv.go:632
	            				pkg/cmd/roachtest/monitor.go:115
	            				external/org_golang_x_sync/errgroup/errgroup.go:78
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	QPS of 645.00 at time 2024-10-31 07:08:37.894839281 +0000 UTC m=+5609.756790770 is below minimum allowable QPS of 1800.00
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
	            	  | 	pkg/cmd/roachtest/tests/kv.go:615
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
	            	  | 	pkg/cmd/roachtest/tests/kv.go:632
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:115
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) QPS of 645.00 at time 2024-10-31 07:08:37.894839281 +0000 UTC m=+5609.756790770 is below minimum allowable QPS of 1800.00
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2386).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

#133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3]
#130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

_{This test on roachdash | Improve this report!}

Oct 31 '24 07:10 cockroach-teamcity

cockroach cockroach copied to clipboard

roachtest: kv/gracefuldraining failed

cockroach
cockroach copied to clipboard