roachtest: vm_preemption failed

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ b083c4a9b946e2a7d6a79b46eef77410f0b742ce:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region failed: VMs preempted during the test run: teamcity-15141330-1715060991-93-n8cpu16-geo-0008 (us-west1-b)

**Other Failures:**
(decommissionbench.go:721).runDecommissionBench: monitor failure: full command output in run_141023.948990917_n8_workload-run-tpcc-wa.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region/run_1/ssh/ssh_141023.981003603_n8_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-38519

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test multitenant/distsql/instances=20/bundle=off/timeout=1 failed: VMs preempted during the test run: teamcity-16645770-1724823955-09-n4cpu4-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2298).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): output:: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/multitenant/distsql/instances=20/bundle=off/timeout=1/run_1/ssh/ssh_055423.469189019_n1_find-ports.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=off/timeout=1/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test cdc/initial-scan-only/parquet failed: VMs preempted during the test run: teamcity-16645770-1724823955-25-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cdc.go:378).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan-only/parquet/run_1/ssh/ssh_063028.363716203_n3_sudo-systemctl-stop-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test cdc/mixed-versions failed: VMs preempted during the test run: teamcity-16645770-1724823955-27-n5cpu4-0003 (us-east4-b), teamcity-16645770-1724823955-27-n5cpu4-0004 (us-east4-b), teamcity-16645770-1724823955-27-n5cpu4-0005 (us-east4-b)

**Other Failures:**
(mixedversion.go:710).Run: EOF [owner=test-eng]
(cluster.go:2411).Run: context canceled
test artifacts and logs in: /artifacts/cdc/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16645770-1724823955-32-n6cpu16-0002 (us-central1-a), teamcity-16645770-1724823955-32-n6cpu16-0003 (us-central1-a)

**Other Failures:**
(cluster.go:2411).Run: full command output in run_065724.055357825_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: VMs preempted during the test run: teamcity-16645770-1724823955-21-n9cpu8-0006 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16645770-1724823955-30-n6cpu16-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_073105.527174049_n3_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-16645770-1724823955-45-n8cpu16-geo-0003 (us-west1-b), teamcity-16645770-1724823955-45-n8cpu16-geo-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2411).Run: full command output in run_073452.100577889_n1_cockroach-workload-f.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1/ssh/ssh_073452.148497058_n1_cockroach-workload-f.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test costfuzz/seed-multi-region failed: VMs preempted during the test run: teamcity-16645770-1724823955-41-n9cpu4-geo-0002 (us-west1-b)

**Other Failures:**
(cluster.go:2348).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/costfuzz/seed-multi-region/run_1/ssh/ssh_075803.847646325_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/costfuzz/seed-multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test clearrange/checks=true failed: VMs preempted during the test run: teamcity-16645770-1724823955-37-n10cpu16-0009 (us-east1-c), teamcity-16645770-1724823955-37-n10cpu16-0008 (us-east1-c)

**Other Failures:**
(cluster.go:2411).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/clearrange/checks=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-16645770-1724823955-53-n10cpu2-0004 (us-central1-a), teamcity-16645770-1724823955-53-n10cpu2-0009 (us-central1-a), teamcity-16645770-1724823955-53-n10cpu2-0001 (us-central1-a)

**Other Failures:**
(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/failover.go:260
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/failover.go:90
	            				main/pkg/cmd/roachtest/test_runner.go:1223
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	pq: server is not accepting clients, try another node
	Test:       	failover/chaos/read-write/lease=expiration
(require.go:1357).NoError: FailNow called
(cluster.go:2411).Run: context canceled
(cluster.go:2411).Run: context canceled
(cluster.go:2411).Run: context canceled
(cluster.go:2411).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16645770-1724823955-14-n8cpu8-0001 (us-east1-d)

**Other Failures:**
(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
	            				main/pkg/cmd/roachtest/test_runner.go:1223
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	monitor failure: context canceled
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | [...repeated from below...]
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).wait
	            	  | 	main/pkg/cmd/roachtest/monitor.go:237
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1223
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (4) context canceled
	            	  | EOF [owner=test-eng]
	            	    └─ Wraps: (5) EOF [owner=test-eng]
	            	    └─ Wraps: (6) context canceled
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
	Test:       	allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test kv95/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-16645770-1724823955-93-n2cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2411).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test kv0/enc=false/nodes=3/cpu=96 failed: VMs preempted during the test run: teamcity-16645770-1724823955-87-n4cpu96-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2411).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=96/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=96
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: VMs preempted during the test run: teamcity-16645770-1724823955-100-n4cpu32-0003 (us-east1-b), teamcity-16645770-1724823955-100-n4cpu32-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2298).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/cpu=32/size=64kb/run_1/ssh/ssh_100426.528753150_n1_init-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/size=64kb/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test kv95/enc=false/nodes=3/cpu=32/seq failed: VMs preempted during the test run: teamcity-16645770-1724823955-86-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2411).Run: full command output in run_095818.479462377_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/cpu=32/seq/run_1/ssh/ssh_095818.506252214_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/seq/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test tpcc/w=100/nodes=3/chaos=true failed: VMs preempted during the test run: teamcity-16645770-1724823955-80-n4cpu4-0001 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/w=100/nodes=3/chaos=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test replicate/wide failed: VMs preempted during the test run: teamcity-16645770-1724823955-112-n9cpu1-0002 (us-east1-c)

**Other Failures:**
(allocator.go:357).func1: pq: get-system-privileges: replica unavailable: (n1,s1):28 unable to serve request to r54:/Table/5{2-3} [(n9,s9):6, (n6,s6):34, (n1,s1):28, (n2,s2):32, (n7,s7):5, (n8,s8):7, (n5,s5):31, next=35, gen=122]: lost quorum (down: (n9,s9):6,(n2,s2):32,(n7,s7):5,(n8,s8):7); closed timestamp: 1724841483.779064103,0 (2024-08-28 10:38:03); raft status: {"id":"1c","term":15,"vote":"1c","commit":350,"lead":"0","raftState":"StatePreCandidate","applied":350,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal GC [/Table/52,/Table/52/NULL), GC [/Table/52,/Table/52/NULL), GC [/Table/52,/Table/52/NULL), GC [/Table/52,/Table/52/NULL)
test artifacts and logs in: /artifacts/replicate/wide/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=1
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test sysbench/oltp_read_only/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16645770-1724823955-117-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2411).Run: full command output in run_103944.704757269_n4_sysbench-dbdriverpgs.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/sysbench/oltp_read_only/nodes=3/cpu=32/conc=256/run_1/ssh/ssh_103944.741113159_n4_sysbench-dbdriverpgs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/sysbench/oltp_read_only/nodes=3/cpu=32/conc=256/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test tpcc/multiregion/survive=region/chaos=true failed: VMs preempted during the test run: teamcity-16645770-1724823955-119-n10cpu4-geo-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test tpcc/mixed-headroom/n5cpu16 failed: VMs preempted during the test run: teamcity-16645770-1724823955-126-n5cpu16-0001 (us-east1-b)

**Other Failures:**
(mixedversion.go:710).Run: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test ycsb/A/nodes=3 failed: VMs preempted during the test run: teamcity-16645770-1724823955-136-n4cpu8-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2411).Run: full command output in run_110152.872458355_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/A/nodes=3/run_1/ssh/ssh_110152.907994674_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed: VMs preempted during the test run: teamcity-16645770-1724823955-139-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_111939.902942496_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1/ssh/ssh_111939.928733926_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test ycsb/A/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-16645770-1724823955-138-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2411).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-16645770-1724823955-129-n10cpu4-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-16645770-1724823955-130-n12cpu4hm-geo-0002 (us-east1-b), teamcity-16645770-1724823955-130-n12cpu4hm-geo-0003 (us-east1-b), teamcity-16645770-1724823955-130-n12cpu4hm-geo-0001 (us-east1-b), teamcity-16645770-1724823955-130-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 1ba029879bcc4afb64a698b3abfa87d2eff31533:

test schemachange/indexschemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16645770-1724823955-114-n5cpu16-0002 (us-east1-b), teamcity-16645770-1724823955-114-n5cpu16-0005 (us-east1-b), teamcity-16645770-1724823955-114-n5cpu16-0003 (us-east1-b), teamcity-16645770-1724823955-114-n5cpu16-0004 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 03f417ee6d97880b3bc4bc885c2e925649de6283:

test cdc/kafka-oauth failed: VMs preempted during the test run: teamcity-16663383-1724910471-24-n4cpu4-0002 (us-east1-b)

**Other Failures:**
(cdc.go:378).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_061505.531234154_n2_sudo-systemctl-stop-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 03f417ee6d97880b3bc4bc885c2e925649de6283:

test cdc/initial-scan-only/parquet failed: VMs preempted during the test run: teamcity-16663383-1724910471-23-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cdc.go:378).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan-only/parquet/run_1/ssh/ssh_062619.860142678_n3_sudo-systemctl-stop-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 03f417ee6d97880b3bc4bc885c2e925649de6283:

test cdc/initial-scan-only failed: VMs preempted during the test run: teamcity-16663383-1724910471-22-n4cpu16-0001 (us-east1-b), teamcity-16663383-1724910471-22-n4cpu16-0003 (us-east1-b), teamcity-16663383-1724910471-22-n4cpu16-0002 (us-east1-b)

**Other Failures:**
(cdc.go:148).func1: error exporting stats file: full command output in run_064213.362660369_n1_mkdir-p-perf.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan-only/run_1/ssh/ssh_064213.420802350_n1_mkdir-p-perf.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1 @ 03f417ee6d97880b3bc4bc885c2e925649de6283:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16663383-1724910471-13-n8cpu8-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2403).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_065842.786270932_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]
• #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc]
• #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc]
• #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
• #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]
• #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc]
• #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!