origin icon indicating copy to clipboard operation
origin copied to clipboard

OVNK BGP: add more debug information for failed test cases

Open jcaamano opened this issue 5 months ago • 11 comments

jcaamano avatar Jun 04 '25 18:06 jcaamano

/test ?

jcaamano avatar Jun 04 '25 18:06 jcaamano

@jcaamano: The following commands are available to trigger required jobs:

/test e2e-aws-jenkins
/test e2e-aws-ovn-edge-zones
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-image-registry
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-ovn
/test e2e-gcp-ovn-builds
/test e2e-gcp-ovn-image-ecosystem
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test images
/test lint
/test okd-scos-images
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
/test e2e-agnostic-ovn-cmd
/test e2e-aws
/test e2e-aws-csi
/test e2e-aws-disruptive
/test e2e-aws-etcd-certrotation
/test e2e-aws-etcd-recovery
/test e2e-aws-ovn
/test e2e-aws-ovn-cgroupsv2
/test e2e-aws-ovn-etcd-scaling
/test e2e-aws-ovn-ipsec-serial
/test e2e-aws-ovn-kube-apiserver-rollout
/test e2e-aws-ovn-kubevirt
/test e2e-aws-ovn-serial-publicnet-1of2
/test e2e-aws-ovn-serial-publicnet-2of2
/test e2e-aws-ovn-single-node
/test e2e-aws-ovn-single-node-serial
/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-single-node-techpreview-serial
/test e2e-aws-ovn-single-node-upgrade
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-rollback
/test e2e-aws-ovn-upi
/test e2e-aws-ovn-virt-techpreview
/test e2e-aws-proxy
/test e2e-azure
/test e2e-azure-ovn-etcd-scaling
/test e2e-azure-ovn-upgrade
/test e2e-baremetalds-kubevirt
/test e2e-external-aws
/test e2e-external-aws-ccm
/test e2e-external-vsphere-ccm
/test e2e-gcp-csi
/test e2e-gcp-disruptive
/test e2e-gcp-fips-serial-1of2
/test e2e-gcp-fips-serial-2of2
/test e2e-gcp-ovn-etcd-scaling
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-ovn-techpreview
/test e2e-gcp-ovn-techpreview-serial-1of2
/test e2e-gcp-ovn-techpreview-serial-2of2
/test e2e-gcp-ovn-usernamespace
/test e2e-hypershift-conformance
/test e2e-metal-ipi-ovn
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview
/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview
/test e2e-metal-ipi-ovn-dualstack-local-gateway
/test e2e-metal-ipi-ovn-kube-apiserver-rollout
/test e2e-metal-ipi-serial-1of2
/test e2e-metal-ipi-serial-2of2
/test e2e-metal-ipi-serial-ovn-ipv6-1of2
/test e2e-metal-ipi-serial-ovn-ipv6-2of2
/test e2e-metal-ipi-virtualmedia
/test e2e-metal-ovn-single-node-live-iso
/test e2e-metal-ovn-single-node-with-worker-live-iso
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-openstack-ovn
/test e2e-openstack-serial
/test e2e-vsphere-ovn-dualstack-primaryv6
/test e2e-vsphere-ovn-etcd-scaling
/test okd-e2e-gcp
/test okd-scos-e2e-aws-ovn

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-origin-main-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
pull-ci-openshift-origin-main-e2e-agnostic-ovn-cmd
pull-ci-openshift-origin-main-e2e-aws
pull-ci-openshift-origin-main-e2e-aws-csi
pull-ci-openshift-origin-main-e2e-aws-disruptive
pull-ci-openshift-origin-main-e2e-aws-ovn
pull-ci-openshift-origin-main-e2e-aws-ovn-cgroupsv2
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones
pull-ci-openshift-origin-main-e2e-aws-ovn-etcd-scaling
pull-ci-openshift-origin-main-e2e-aws-ovn-fips
pull-ci-openshift-origin-main-e2e-aws-ovn-kube-apiserver-rollout
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift-serial
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-publicnet-1of2
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-publicnet-2of2
pull-ci-openshift-origin-main-e2e-aws-ovn-single-node
pull-ci-openshift-origin-main-e2e-aws-ovn-single-node-serial
pull-ci-openshift-origin-main-e2e-aws-ovn-single-node-upgrade
pull-ci-openshift-origin-main-e2e-aws-ovn-upgrade
pull-ci-openshift-origin-main-e2e-aws-proxy
pull-ci-openshift-origin-main-e2e-azure
pull-ci-openshift-origin-main-e2e-azure-ovn-etcd-scaling
pull-ci-openshift-origin-main-e2e-azure-ovn-upgrade
pull-ci-openshift-origin-main-e2e-gcp-csi
pull-ci-openshift-origin-main-e2e-gcp-disruptive
pull-ci-openshift-origin-main-e2e-gcp-fips-serial-1of2
pull-ci-openshift-origin-main-e2e-gcp-fips-serial-2of2
pull-ci-openshift-origin-main-e2e-gcp-ovn
pull-ci-openshift-origin-main-e2e-gcp-ovn-etcd-scaling
pull-ci-openshift-origin-main-e2e-gcp-ovn-rt-upgrade
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade
pull-ci-openshift-origin-main-e2e-hypershift-conformance
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-dualstack
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-dualstack-local-gateway
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-kube-apiserver-rollout
pull-ci-openshift-origin-main-e2e-metal-ipi-serial-1of2
pull-ci-openshift-origin-main-e2e-metal-ipi-serial-2of2
pull-ci-openshift-origin-main-e2e-metal-ipi-serial-ovn-ipv6-1of2
pull-ci-openshift-origin-main-e2e-metal-ipi-serial-ovn-ipv6-2of2
pull-ci-openshift-origin-main-e2e-metal-ipi-virtualmedia
pull-ci-openshift-origin-main-e2e-openstack-ovn
pull-ci-openshift-origin-main-e2e-openstack-serial
pull-ci-openshift-origin-main-e2e-vsphere-ovn
pull-ci-openshift-origin-main-e2e-vsphere-ovn-dualstack-primaryv6
pull-ci-openshift-origin-main-e2e-vsphere-ovn-etcd-scaling
pull-ci-openshift-origin-main-e2e-vsphere-ovn-upi
pull-ci-openshift-origin-main-images
pull-ci-openshift-origin-main-lint
pull-ci-openshift-origin-main-okd-e2e-gcp
pull-ci-openshift-origin-main-okd-scos-e2e-aws-ovn
pull-ci-openshift-origin-main-okd-scos-images
pull-ci-openshift-origin-main-unit
pull-ci-openshift-origin-main-verify
pull-ci-openshift-origin-main-verify-deps

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Jun 04 '25 18:06 openshift-ci[bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcaamano

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci[bot] avatar Jun 04 '25 18:06 openshift-ci[bot]

/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

jcaamano avatar Jun 05 '25 08:06 jcaamano

/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

jcaamano avatar Jun 05 '25 13:06 jcaamano

/hold

testing that it actually works

jcaamano avatar Jun 05 '25 13:06 jcaamano

/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview

jcaamano avatar Jun 05 '25 18:06 jcaamano

Job Failure Risk Analysis for sha: ca503b1e0e9af6f9b32d7f61678b581c34b52f08

Job Name Failure Risk
pull-ci-openshift-origin-main-e2e-aws-disruptive Medium
[sig-node] static pods should start after being created
Potential external regression detected for High Risk Test analysis

Open Bugs
[sig-node] static pods should start after being created
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift IncompleteTests
Tests for this run (23) are below the historical average (745): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift-serial IncompleteTests
Tests for this run (23) are below the historical average (386): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-azure-ovn-etcd-scaling Low
[bz-Cloud Compute] clusteroperator/control-plane-machine-set should not change condition/Degraded
This test has passed 0.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:azure SecurityMode:default Topology:ha Upgrade:none] in the last week.
---
[bz-kube-storage-version-migrator] clusteroperator/kube-storage-version-migrator should not change condition/Available
This test has passed 0.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:azure SecurityMode:default Topology:ha Upgrade:none] in the last week.

Open Bugs
[CI] e2e-openstack-ovn-etcd-scaling job permanent fails at many openshift-test tests
pull-ci-openshift-origin-main-e2e-gcp-ovn-etcd-scaling Low
[bz-Cloud Compute] clusteroperator/control-plane-machine-set should not change condition/Degraded
This test has passed 0.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:gcp SecurityMode:default Topology:ha Upgrade:none] in the last week.
pull-ci-openshift-origin-main-e2e-gcp-ovn-rt-upgrade IncompleteTests
Tests for this run (32) are below the historical average (1729): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt[bot] avatar Jun 06 '25 02:06 openshift-trt[bot]

/hold cancel

jcaamano avatar Jun 06 '25 09:06 jcaamano

/retitle NO-JIRA: OVNK BGP: add more debug information for failed test cases

jcaamano avatar Jun 18 '25 14:06 jcaamano

@jcaamano: This pull request explicitly references no jira issue.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Jun 18 '25 14:06 openshift-ci-robot

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot avatar Sep 17 '25 01:09 openshift-bot

Job Failure Risk Analysis for sha: 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1

Job Name Failure Risk
pull-ci-openshift-origin-main-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback MissingData
pull-ci-openshift-origin-main-e2e-aws-ovn Medium
[sig-instrumentation] Metrics should grab all metrics from kubelet /metrics/resource endpoint [Suite:openshift/conformance/parallel] [Suite:k8s]
This test has passed 94.28% of 1854 runs on release 4.20 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-aws-ovn-cgroupsv2 Medium
[sig-instrumentation] Metrics should grab all metrics from kubelet /metrics/resource endpoint [Suite:openshift/conformance/parallel] [Suite:k8s]
This test has passed 94.28% of 1854 runs on release 4.20 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-aws-ovn-etcd-scaling Low
[bz-Cloud Compute] clusteroperator/control-plane-machine-set should not change condition/Degraded
This test has passed 50.00% of 2 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:aws SecurityMode:default Topology:ha Upgrade:none] in the last week.
pull-ci-openshift-origin-main-e2e-aws-proxy Medium
[sig-node] Pods Extended Pod Container lifecycle evicted pods should be terminal [Suite:openshift/conformance/parallel] [Suite:k8s]
This test has passed 95.61% of 1801 runs on release 4.20 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-azure-ovn-etcd-scaling High
[sig-architecture] platform pods in ns/openshift-etcd should not exit an excessive amount of times
This test has passed 100.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:azure SecurityMode:default Topology:ha Upgrade:none] in the last week.
---
[bz-etcd][invariant] alert/etcdMembersDown should not be at or above info
This test has passed 100.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:azure SecurityMode:default Topology:ha Upgrade:none] in the last week.
pull-ci-openshift-origin-main-e2e-azure-ovn-upgrade IncompleteTests
pull-ci-openshift-origin-main-e2e-gcp-disruptive High
[sig-network] pods should successfully create sandboxes by writing network status
This test has passed 98.94% of 3489 runs on release 4.20 [Overall] in the last week.

Open Bugs
etcd timeouts causing failed pod sandbox creation writing network status
---
[bz-Monitoring] clusteroperator/monitoring should not change condition/Degraded
This test has passed 98.57% of 3489 runs on release 4.20 [Overall] in the last week.
---
[Jira:"kube-apiserver"] monitor test audit-log-analyzer collection
This test has passed 98.05% of 3489 runs on release 4.20 [Overall] in the last week.
---
[Jira:"Node / Kubelet"] monitor test pod-lifecycle test evaluation
This test has passed 98.08% of 3489 runs on release 4.20 [Overall] in the last week.
---
Showing 4 of 11 test results
pull-ci-openshift-origin-main-e2e-vsphere-ovn-etcd-scaling High
API LBs follow /readyz of kube-apiserver and stop sending requests before server shutdowns for external clients
This test has passed 99.66% of 3489 runs on release 4.20 [Overall] in the last week.

Open Bugs
[CI] e2e-openstack-ovn-etcd-scaling job permanent fails at many openshift-test tests
---
[bz-openshift-apiserver] clusteroperator/openshift-apiserver should not change condition/Available
This test has passed 98.25% of 3489 runs on release 4.20 [Overall] in the last week.

Open Bugs
CI: API is broken in periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-single-node-techpreview-serial

openshift-trt[bot] avatar Oct 03 '25 19:10 openshift-trt[bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot avatar Nov 17 '25 00:11 openshift-bot

@jcaamano: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp-techpreview 4f6985ec13683419bead2f0a757efe66a322afd5 link false /test e2e-metal-ipi-ovn-dualstack-bgp-techpreview
ci/prow/e2e-aws-ovn-etcd-scaling 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-aws-disruptive 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-aws-disruptive
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview ca503b1e0e9af6f9b32d7f61678b581c34b52f08 link false /test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview
ci/prow/okd-e2e-gcp 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test okd-e2e-gcp
ci/prow/e2e-aws-ovn-single-node-serial 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-aws-ovn-single-node-serial
ci/prow/e2e-gcp-fips-serial-1of2 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-gcp-fips-serial-1of2
ci/prow/e2e-vsphere-ovn-upi 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link true /test e2e-vsphere-ovn-upi
ci/prow/e2e-gcp-ovn 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link true /test e2e-gcp-ovn
ci/prow/e2e-gcp-fips-serial-2of2 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-gcp-fips-serial-2of2
ci/prow/e2e-openstack-ovn 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-openstack-ovn
ci/prow/e2e-azure-ovn-etcd-scaling 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-openstack-serial 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-openstack-serial
ci/prow/e2e-gcp-ovn-etcd-scaling 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-gcp-ovn-etcd-scaling
ci/prow/okd-scos-e2e-aws-ovn 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-single-node-upgrade 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-aws-ovn-single-node-upgrade
ci/prow/e2e-vsphere-ovn-etcd-scaling 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-azure-ovn-upgrade 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-azure-ovn-upgrade
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-vsphere-ovn-dualstack-primaryv6
ci/prow/e2e-aws-ovn-serial-publicnet-1of2 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-aws-ovn-serial-publicnet-1of2
ci/prow/e2e-gcp-ovn-rt-upgrade 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-gcp-ovn-rt-upgrade
ci/prow/e2e-gcp-disruptive 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test e2e-gcp-disruptive
ci/prow/4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link false /test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
ci/prow/e2e-aws-csi 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link true /test e2e-aws-csi
ci/prow/e2e-gcp-csi 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link true /test e2e-gcp-csi
ci/prow/go-verify-deps 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link true /test go-verify-deps
ci/prow/e2e-metal-ipi-ovn-ipv6 1e3ce0d3f0d05f0a90f4c9b4625f42187fda5fe1 link true /test e2e-metal-ipi-ovn-ipv6

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci[bot] avatar Nov 18 '25 13:11 openshift-ci[bot]