cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

roachtest: cdc/kafka-chaos-multiple-sink-nodes failed

Open cockroach-teamcity opened this issue 1 year ago • 1 comments

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

(monitor.go:149).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 5m0.039207138s vs 5m0s
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]
/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-43382

cockroach-teamcity avatar Oct 19 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 20 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

(monitor.go:149).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 5m0.115342607s vs 5m0s
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 21 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 330f659a3f244d1386bb86c89a258d2da6063959:

(monitor.go:149).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 5m0.731542046s vs 5m0s
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 22 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ f4bf28879d49979d438967379b44c2fe3464f968:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 23 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 24 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 047a8e99eb7a35e3f13ecf6b113f1965f206e3b9:

(monitor.go:149).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 5m0.047403425s vs 5m0s
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 25 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 3839fb143f98e5009a994ad58fb6b82dc7badc55:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 26 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 3839fb143f98e5009a994ad58fb6b82dc7badc55:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 27 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 3839fb143f98e5009a994ad58fb6b82dc7badc55:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 28 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ 416acdb7560e4548f3b82c5a522ef48592471c7d:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 29 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ c357ebbfdd72bb6aecef57c84850845b5955382f:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #127110 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 30 '24 10:10 cockroach-teamcity

roachtest.cdc/kafka-chaos-multiple-sink-nodes failed with artifacts on release-24.3 @ b1fbd0c00f283c147b0c3a36a8f25ce7eec41dff:

(cdc.go:394).Close: error shutting down prometheus/grafana: COMMAND_PROBLEM: exit status 7
test artifacts and logs in: /artifacts/cdc/kafka-chaos-multiple-sink-nodes/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #133923 roachtest: cdc/kafka-chaos-multiple-sink-nodes failed [A-cdc C-test-failure O-roachtest O-robot T-cdc branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 31 '24 09:10 cockroach-teamcity

Should be fixed by #133904 and #133993

andyyang890 avatar Oct 31 '24 19:10 andyyang890