cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

roachtest: multitenant-upgrade failed

Open cockroach-teamcity opened this issue 1 year ago • 6 comments

roachtest.multitenant-upgrade failed with artifacts on master @ b2e31876366324c2ebe5c2ad8bbd644997e90864:

(multitenant_utils.go:235).func1: full command output in run_155620.731165053_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/multi-tenant

This test on roachdash | Improve this report!

Jira issue: CRDB-36016

cockroach-teamcity avatar Feb 10 '24 15:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ b2e31876366324c2ebe5c2ad8bbd644997e90864:

(multitenant_utils.go:235).func1: full command output in run_184759.364805790_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 12 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 814a375d4c0e79d875c42452725f05f6c27294e3:

(multitenant_utils.go:235).func1: full command output in run_185301.084551217_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 13 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 0b7ae19e2b94b851ed8812914f57032aab699811:

(multitenant_utils.go:235).func1: full command output in run_183158.357020432_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 17 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a:

(multitenant_utils.go:235).func1: full command output in run_182121.367336037_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 18 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ e50b0ec4d3a53f81e26f3776ae3f3be55d435a9a:

(multitenant_utils.go:235).func1: full command output in run_181622.591568851_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 21 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ d4306a9e895eaf50ef9b87b4ba09581447f7dd70:

(multitenant_utils.go:235).func1: full command output in run_182744.285218650_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 23 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 24414b864f65c6544ea987f7970fa1c192ca3310:

(multitenant_utils.go:235).func1: full command output in run_180004.922025396_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 24 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ c9c3cc5f3c3a4a6ab556f4b9d5b6ec0381901bdb:

(multitenant_utils.go:235).func1: full command output in run_181659.130031294_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 26 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 9a494f4e4846a67ceea18c4b4993cefbeb19c6d9:

(multitenant_utils.go:243).func1: full command output in run_184044.702058690_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2330).Run: context canceled
(cluster.go:2330).Run: context canceled
(cluster.go:2330).Run: context canceled
(cluster.go:2330).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Feb 29 '24 18:02 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 791969fa98711e681289fa4f61f3d58c8fe44263:

(multitenant_utils.go:243).func1: full command output in run_185631.973709802_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2330).Run: context canceled
(cluster.go:2330).Run: context canceled
(cluster.go:2330).Run: context canceled
(cluster.go:2330).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 02 '24 18:03 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 559994e230ac3776dd1188e66678cce9205ab345:

(multitenant_utils.go:243).func1: full command output in run_141122.100485855_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2329).Run: context canceled
(cluster.go:2329).Run: context canceled
(cluster.go:2329).Run: context canceled
(cluster.go:2329).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 08 '24 14:03 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 72646a555214c0705781e440b9df585d5eea9511:

(multitenant_utils.go:243).func1: full command output in run_184455.217604409_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2329).Run: context canceled
(cluster.go:2329).Run: context canceled
(cluster.go:2329).Run: context canceled
(cluster.go:2329).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 09 '24 18:03 cockroach-teamcity

Can anyone provide context on this and why it's a SQL Queries issue?

mgartner avatar Mar 13 '24 16:03 mgartner

Adam bisected the failure to a change I made here

yuzefovich avatar Mar 13 '24 16:03 yuzefovich

roachtest.multitenant-upgrade failed with artifacts on master @ ebf92d22a7e4afe2423e681d49c615104b5f0098:

(multitenant_utils.go:236).func1: full command output in run_144127.712532843_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 15 '24 14:03 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 34caf9d2c267f95f6aef2713708806e6f14948d6:

(multitenant_utils.go:236).func1: full command output in run_162645.426803295_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 16 '24 16:03 cockroach-teamcity

roachtest.multitenant-upgrade failed with artifacts on master @ 067e48d29b9093038f6fcf2074cd761ffdcd4fe2:

(multitenant_utils.go:236).func1: full command output in run_153750.310648765_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
(cluster.go:2332).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 17 '24 15:03 cockroach-teamcity

Hi @yuzefovich, is there any update on this issue? Thanks!

rytaft avatar Mar 20 '24 15:03 rytaft

roachtest.multitenant-upgrade failed with artifacts on master @ f0116ea373a2b87155e7f0264df4f783ce177360:

(multitenant_utils.go:238).func1: full command output in run_151840.502319591_n2_COCKROACHTESTINGFORC.log: COMMAND_PROBLEM: exit status 1
(cluster.go:2343).Run: context canceled
(cluster.go:2343).Run: context canceled
(cluster.go:2343).Run: context canceled
(cluster.go:2343).Run: context canceled
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Mar 20 '24 15:03 cockroach-teamcity

I'm reverting the change that Adam bisected to and downgrading this to GA-blocker. I do want to understand why this test started to fail (which can be done during stability period), but it's not super urgent to do so - if I don't get to it during stability, reverting #118287 restores the status quo that we've had for many years.

yuzefovich avatar Mar 20 '24 20:03 yuzefovich

Downgrading this to P-3 based on the previous comment.

rytaft avatar Mar 27 '24 15:03 rytaft

I finally got around to looking into this issue, and it has actually been resolved on master. I bisected the fix to be #123520. In other words, #118287 tickled a real bug in mixed-version 23.2-24.1 multi-tenant environment due to a backwards-incompatible change merged during 24.1 development cycle. Kudos to @renatolabs for tracking it down!

yuzefovich avatar May 29 '24 03:05 yuzefovich