cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard
verified publisher icon cockroachdb

roachtest: c2c/mixed-version failed

Open cockroach-teamcity opened this issue 2 months ago • 5 comments

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/mixed-version failed with artifacts on release-25.2 @ 58e26b796725ea942bf2e1c23dfc762ae40f3896:

(mixedversion.go:810).Run: dest: mixed-version test failure while running step 9 (restart node 8 with binary version release-25.2): failed to wait for replication after starting cockroach: failed to wait for SQL to be ready: read tcp 172.17.0.3:44410 -> 34.23.149.24:26257: i/o timeout
(mixedversion.go:810).Run: source: preparing to run step 14: failed to get binary version for node 4 (system): context canceled
test artifacts and logs in: /artifacts/c2c/mixed-version/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicBufferedSender=false
  • mvtDeploymentMode=system-only
  • mvtVersions=v25.1.10 → release-25.2
  • runtimeAssertionsBuild=true
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #155697 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-25.4.0-rc]
  • #154125 roachtest: c2c/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-25.4 sync-me target-release-25.4.1]
  • #153504 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.3.2-rc]
  • #153495 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-24.3.20-rc]
  • #150401 roachtest: c2c/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-master]
  • #150047 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.3]
/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-55611

cockroach-teamcity avatar Oct 21 '25 08:10 cockroach-teamcity

This is another one of the pgwire error loop failures. Looking at the stacks, this seems to be related to leasing. All of the requests are attempting to acquire a lease on the system db.

Image

jeffswenson avatar Oct 21 '25 15:10 jeffswenson

waiting for released versions to catch up with the patch

dt avatar Oct 27 '25 18:10 dt

roachtest.c2c/mixed-version failed with artifacts on release-25.2 @ 2e2d5cf50d9e9645d7c43d531b5dd667096989aa:

(mixedversion.go:810).Run: dest: preparing to run step 12: failed to get binary version for node 7 (system): pq: replica unavailable: (n3,s3):2 unable to serve request to r33:/NamespaceTable/{30-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: closed timestamp: 1764233212.066652636,0 (2025-11-27 08:46:52); raft status: {"id":"2","term":8,"vote":"1","commit":39,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":39,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
(mixedversion.go:810).Run: source: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/c2c/mixed-version/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicBufferedSender=true
  • mvtDeploymentMode=system-only
  • mvtVersions=v25.1.10 → release-25.2
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #155697 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-25.4.0-rc]
  • #150401 roachtest: c2c/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Nov 27 '25 08:11 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/mixed-version failed with artifacts on release-25.2 @ a22a1998b1072de08871a02f28316cb25855eee1:

(mixedversion.go:810).Run: dest: mixed-version test failure while running step 9 (restart node 7 with binary version release-25.2): failed to wait for replication after starting cockroach: failed to wait for SQL to be ready: read tcp 172.17.0.3:56522 -> 35.196.121.143:26257: i/o timeout
(mixedversion.go:810).Run: source: preparing to run step 27: failed to get binary version for node 4 (system): context canceled
test artifacts and logs in: /artifacts/c2c/mixed-version/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicBufferedSender=true
  • mvtDeploymentMode=system-only
  • mvtVersions=v25.1.10 → release-25.2
  • runtimeAssertionsBuild=true
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #158460 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-24.3.24-rc release-blocker]
  • #155697 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-25.4.0-rc]
  • #150401 roachtest: c2c/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Dec 02 '25 09:12 cockroach-teamcity

roachtest.c2c/mixed-version failed with artifacts on release-25.2 @ 22a273b480062c8b0937bec395fedc750794cc7d:

(mixedversion.go:810).Run: dest: unexpected node event: n6: cockroach process for system interface died (exit code 1)
(mixedversion.go:810).Run: source: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/c2c/mixed-version/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicBufferedSender=true
  • mvtDeploymentMode=system-only
  • mvtVersions=v24.3.23 → release-25.2
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #158970 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-25.3.6-rc]
  • #158944 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-25.3]
  • #158460 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-24.3.24-rc release-blocker]
  • #155697 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-25.4.0-rc]
  • #150401 roachtest: c2c/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Dec 11 '25 08:12 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/mixed-version failed with artifacts on release-25.2 @ a7431d38d7afd3d91f277fca842284ce006ac433:

(mixedversion.go:810).Run: dest: mixed-version test failure while running step 28 (run "cutover and allow source to finalize"): job 1132274460539912193 did not reach state running after 1m0s
(mixedversion.go:810).Run: source: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/c2c/mixed-version/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicBufferedSender=true
  • mvtDeploymentMode=system-only
  • mvtVersions=v25.1.10 → release-25.2
  • runtimeAssertionsBuild=true
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #158970 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-25.3.6-rc]
  • #158944 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-25.3]
  • #158460 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-24.3.24-rc release-blocker]
  • #155697 roachtest: c2c/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-25.4.0-rc]
  • #150401 roachtest: c2c/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Dec 13 '25 08:12 cockroach-teamcity