cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

roachtest: cdc/pubsub-sink/assume-role failed

Open cockroach-teamcity opened this issue 3 years ago • 11 comments
trafficstars

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 0352fbee5e6933e8bca28aae4b136bcd495ab227:

		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_073444.689975139_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  | I220731 07:34:45.911560 1 workload/cli/run.go:427  [-] 1  creating load generator...
		  | I220731 07:34:45.913945 64 workload/pgx_helpers.go:79  [-] 2  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.14:26257: connect: connection refused]
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.14 user=root database=tpcc``: dial error (dial tcp 10.142.0.14:26257: connect: connection refused)
		  |
		  | stdout:
		  | Initializing 2 connections...
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,cdc.go:287,cdc.go:829,test_runner.go:896: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:829
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6222
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-18212

Epic CRDB-11732

cockroach-teamcity avatar Jul 31 '22 07:07 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ b83e83b7e54fabf6ebd3a42cff67f4b8c5b7b863:

		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_074417.261515267_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  | I220801 07:44:18.406802 1 workload/cli/run.go:427  [-] 1  creating load generator...
		  | I220801 07:44:18.408885 111 workload/pgx_helpers.go:79  [-] 2  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.56:26257: connect: connection refused]
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.56 user=root database=tpcc``: dial error (dial tcp 10.142.0.56:26257: connect: connection refused)
		  |
		  | stdout:
		  | Initializing 2 connections...
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,cdc.go:287,cdc.go:829,test_runner.go:896: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:829
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6222
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 01 '22 07:08 cockroach-teamcity

"failed to initialize the load generator" is a test flake, not an a real failure, so I'll remove the release-blocker tag, but we should still triage this.

HonoreDB avatar Aug 01 '22 14:08 HonoreDB

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 773f7d4445ce3e0e806b7a182adba70a0f270f19:

		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_071113.719576588_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  | I220812 07:11:14.923700 1 workload/cli/run.go:427  [-] 1  creating load generator...
		  | I220812 07:11:14.926801 116 workload/pgx_helpers.go:79  [-] 2  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.132:26257: connect: connection refused]
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.132 user=root database=tpcc``: dial error (dial tcp 10.142.0.132:26257: connect: connection refused)
		  |
		  | stdout:
		  | Initializing 2 connections...
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,cdc.go:287,cdc.go:829,test_runner.go:896: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:829
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 12 '22 07:08 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 5c2c62ecf1bea60c807edc6b4da22d900ad4ae03:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1597
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:177
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_072421.278700404_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:287,cdc.go:816,test_runner.go:896: monitor failure: monitor task failed: dial tcp 34.138.240.36:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:816
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.138.240.36:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 18 '22 07:08 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ c080f7ac2f5be11aedd787dceba77cca7df18a16:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1602
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:176
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_072358.006431375_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:897: monitor failure: monitor task failed: dial tcp 35.237.211.226:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.237.211.226:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 02 '22 07:09 cockroach-teamcity

roachtest.cdc/pubsub-sink failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:

		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     3.0s        0            3.0            1.0     65.0     71.3     71.3     71.3 orderStatus
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     3.0s        0            1.0            0.3     60.8     60.8     60.8     60.8 stockLevel
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:300,cdc.go:794,test_runner.go:917: monitor failure: monitor task failed: read tcp 172.17.0.3:41914 -> 34.138.23.29:26257: read: connection reset by peer
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func8
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:794
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) read tcp 172.17.0.3:41914 -> 34.138.23.29:26257
		Wraps: (6) read
		Wraps: (7) connection reset by peer
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #87683 roachtest: cdc/pubsub-sink failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]
  • #79108 roachtest: cdc/pubsub-sink failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 12 '22 07:09 cockroach-teamcity

roachtest.cdc/pubsub-sink failed with artifacts on master @ 9a05046ce19e7678340e82c70d61e928be95bc72:

		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     3.0s        0            1.0            0.3     71.3     71.3     71.3     71.3 delivery
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     4.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 delivery
		  |     4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     4.0s        0            4.0            1.0     16.8     35.7     35.7     35.7 payment
		  |     4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:300,cdc.go:794,test_runner.go:917: monitor failure: monitor task failed: read tcp 172.17.0.3:48362 -> 34.148.95.11:26257: read: connection reset by peer
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func8
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:794
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) read tcp 172.17.0.3:48362 -> 34.148.95.11:26257
		Wraps: (6) read
		Wraps: (7) connection reset by peer
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #87899 roachtest: cdc/pubsub-sink failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]
  • #79108 roachtest: cdc/pubsub-sink failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 16 '22 07:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 5862e9b3c1e24b3f644716789c96108a8e92fd71:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:917: monitor failure: monitor task failed: dial tcp 34.138.103.200:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.138.103.200:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1048,test_runner.go:947: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 20 '22 17:09 cockroach-teamcity

Looks like I introduced a new bug while fixing the previous one, fixing now.

HonoreDB avatar Sep 20 '22 19:09 HonoreDB

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ bbcb10f982e47943de63a3a2d1faf10d3eb49e75:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.139.149.103:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.139.149.103:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 21 '22 17:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 726cf22b9f06b766d857b4617dec0df18d1e5cd0:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.148.134.64:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.148.134.64:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #88369 roachtest: cdc/pubsub-sink/assume-role failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 22 '22 16:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 34dc56fbb5789b39be47b110bf22332c7f5654f6:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 35.229.118.106:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.229.118.106:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #88369 roachtest: cdc/pubsub-sink/assume-role failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 23 '22 17:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 89f4ad907a1756551bd6864c3e8516eeff6b0e0a:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 35.185.108.152:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.185.108.152:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #88369 roachtest: cdc/pubsub-sink/assume-role failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 24 '22 17:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 51c8aae748d338549400c047796c6c9b892527da:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 35.196.58.108:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.196.58.108:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #88369 roachtest: cdc/pubsub-sink/assume-role failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 25 '22 17:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ ff13325e9368c4e8dd9a4d5cf4aa2ad2f33e9ac0:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.74.155.56:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.74.155.56:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #88740 roachtest: cdc/pubsub-sink/assume-role failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 27 '22 02:09 cockroach-teamcity

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ a0bfa6dafcc206301d3a21887c374db63b377075:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.139.191.22:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.139.191.22:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

  • #88740 roachtest: cdc/pubsub-sink/assume-role failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 27 '22 17:09 cockroach-teamcity