cockroach
cockroach copied to clipboard
roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ d2a5af43be05ade40531e6e0ef23c5880faa4494:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 35.231.108.147:29002: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
This test on roachdash | Improve this report!
Jira issue: CRDB-39846
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 047a7ed79756eef53b8b9ab4c9dd9c5a463496c9:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1247
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 35.185.15.229:29002: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1247
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 35.237.37.137:29002: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1247
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.138.149.104:29002: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 9e1cd533828b7887a48db8635f705669287cf2d6:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.74.235.10:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 8f72a01e81d4751c6615a9a864b8e32076565b09:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.44.51.248:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/cpu_arch=arm64/run_1
Parameters:
ROACHTEST_arch=arm64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 295d09a88895a69e5cc9149fb8165acf78e39e61:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.23.181.197:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ c557fb59f6aec659d364e9002fc083c59c6392b6:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.23.127.167:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 485975b3a824c68c07340e6a336c7864c00d3c6d:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 35.185.75.152:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 485975b3a824c68c07340e6a336c7864c00d3c6d:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.139.201.152:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 6b22014dec8c5e8e6006d710ed05bab5b01b4ae2:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 35.237.91.104:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 0b918d1dc3a9ce1f04975202be3b04ec375a816e:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.23.102.168:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ b78574b4b75b6f9ab8841d96f350697030f5e4df:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 34.74.151.98:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 3afb6935d0ef3de7c0d44cfb3cd54f312752c186:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 104.196.149.188:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
Same failure on other branches
- #126979 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-24.2 release-blocker]
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ dc3ebba01015d2c7275aa27af2a54fe88b87e901:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 35.196.68.247:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
Same failure on other branches
- #126979 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-24.2 release-blocker]
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 67aa2613446c3c6b53bad21f01286c98290cc0a3:
(test_runner.go:1301).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
Parameters:
ROACHTEST_arch=amd64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
Same failure on other branches
- #126979 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-24.2 release-blocker]
I looked briefly at this because it happened to be the first timing out test that Side-Eye automatically captured a snapshot for.
The test seems to poll for the physical planner to get to a point where it uses all the nodes for some queries, and this condition seems to never be satisfied. The test keeps printing:
2024/07/15 09:08:32 multitenant_distsql.go:161: Only 7 nodes present: (1,4,11,15,16,18,20)
First of all, I think the test intended to fail in a better way in such cases, but I think the respective code is broken (or, at least, it doesn't make sense to me). Around here the test seems to want to implement its own 180s timeout. But notice that attempts is only decremented if err != nil. Shouldn't it always be incremented?
There's a similar dubious pattern further down where a different attempts is only decremented on an error.
Now for why the test really fails, I've got no idea. But Side-Eye shows that all the INSERT INTO t SELECT $1,generate_series(1,100)+$2*100,repeat('asdfasdf',1024) queries running at the time of the timeout were blocked on tenantcostclient.(*limiter).Wait() (see e.g.). Could admission control be blocking the writing of the data that the test is depending on?
cc @mgartner @yuzefovich
roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 73ecc8b494c4e7f757bb35d68b3587a51af93b02:
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
main/pkg/cmd/roachtest/test_runner.go:1272
src/runtime/asm_amd64.s:1695
Error: Error message not equal:
expected: "pq: query execution canceled due to statement timeout"
actual : "dial tcp 104.155.166.235:29000: connect: connection refused"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/cpu_arch=arm64/run_1
Parameters:
ROACHTEST_arch=arm64ROACHTEST_cloud=gceROACHTEST_coverageBuild=falseROACHTEST_cpu=4ROACHTEST_encrypted=falseROACHTEST_metamorphicBuild=falseROACHTEST_ssd=0
Let's assume #127219 is the fix and close this issue.
We do have #121260 to track improving the test overall - thanks @andreimatei for your insight.