cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed

Open cockroach-teamcity opened this issue 1 year ago • 3 comments
trafficstars

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ d2a5af43be05ade40531e6e0ef23c5880faa4494:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 35.231.108.147:29002: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

Jira issue: CRDB-39846

cockroach-teamcity avatar Jun 27 '24 06:06 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 047a7ed79756eef53b8b9ab4c9dd9c5a463496c9:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1247
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 35.185.15.229:29002: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 29 '24 06:06 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1247
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 35.237.37.137:29002: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 30 '24 06:06 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1247
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.138.149.104:29002: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 01 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 9e1cd533828b7887a48db8635f705669287cf2d6:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.74.235.10:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 02 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 8f72a01e81d4751c6615a9a864b8e32076565b09:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.44.51.248:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 04 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 295d09a88895a69e5cc9149fb8165acf78e39e61:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.23.181.197:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 05 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ c557fb59f6aec659d364e9002fc083c59c6392b6:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.23.127.167:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 06 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 485975b3a824c68c07340e6a336c7864c00d3c6d:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 35.185.75.152:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 07 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 485975b3a824c68c07340e6a336c7864c00d3c6d:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.139.201.152:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 08 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 6b22014dec8c5e8e6006d710ed05bab5b01b4ae2:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 35.237.91.104:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 09 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 0b918d1dc3a9ce1f04975202be3b04ec375a816e:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.23.102.168:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 10 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ b78574b4b75b6f9ab8841d96f350697030f5e4df:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 34.74.151.98:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 11 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 3afb6935d0ef3de7c0d44cfb3cd54f312752c186:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 104.196.149.188:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #126979 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-24.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 12 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ dc3ebba01015d2c7275aa27af2a54fe88b87e901:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1248
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 35.196.68.247:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #126979 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-24.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 13 '24 06:07 cockroach-teamcity

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 67aa2613446c3c6b53bad21f01286c98290cc0a3:

(test_runner.go:1301).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #126979 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-24.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 15 '24 09:07 cockroach-teamcity

I looked briefly at this because it happened to be the first timing out test that Side-Eye automatically captured a snapshot for.

The test seems to poll for the physical planner to get to a point where it uses all the nodes for some queries, and this condition seems to never be satisfied. The test keeps printing:

2024/07/15 09:08:32 multitenant_distsql.go:161: Only 7 nodes present: (1,4,11,15,16,18,20)

First of all, I think the test intended to fail in a better way in such cases, but I think the respective code is broken (or, at least, it doesn't make sense to me). Around here the test seems to want to implement its own 180s timeout. But notice that attempts is only decremented if err != nil. Shouldn't it always be incremented? There's a similar dubious pattern further down where a different attempts is only decremented on an error.

Now for why the test really fails, I've got no idea. But Side-Eye shows that all the INSERT INTO t SELECT $1,generate_series(1,100)+$2*100,repeat('asdfasdf',1024) queries running at the time of the timeout were blocked on tenantcostclient.(*limiter).Wait() (see e.g.). Could admission control be blocking the writing of the data that the test is depending on?

cc @mgartner @yuzefovich

andreimatei avatar Jul 15 '24 21:07 andreimatei

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on master @ 73ecc8b494c4e7f757bb35d68b3587a51af93b02:

(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1272
	            				src/runtime/asm_amd64.s:1695
	Error:      	Error message not equal:
	            	expected: "pq: query execution canceled due to statement timeout"
	            	actual  : "dial tcp 104.155.166.235:29000: connect: connection refused"
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 16 '24 06:07 cockroach-teamcity

Let's assume #127219 is the fix and close this issue.

mgartner avatar Jul 17 '24 16:07 mgartner

We do have #121260 to track improving the test overall - thanks @andreimatei for your insight.

yuzefovich avatar Jul 17 '24 16:07 yuzefovich