origin icon indicating copy to clipboard operation
origin copied to clipboard

OCPBUGS-39030: kube-apiserver rollout: wait for stability before exiting

Open stbenjam opened this issue 1 year ago • 22 comments
trafficstars

We are sometimes leaving this test with kube-apiserver operator "Progressing" -- this triggers a failure in gather-extra of the "operator conditions kube-apiserver" test, which expects operators to be stable when the job finishes.

Before we return from this test, make sure kube-apiserver is not progressing.

stbenjam avatar Aug 27 '24 15:08 stbenjam

/test e2e-metal-ovn-kube-apiserver-rollout

stbenjam avatar Aug 27 '24 15:08 stbenjam

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

openshift-ci[bot] avatar Aug 27 '24 15:08 openshift-ci[bot]

@stbenjam: The /test command needs one or more targets. The following commands are available to trigger required jobs:

  • /test e2e-aws-jenkins
  • /test e2e-aws-ovn-edge-zones
  • /test e2e-aws-ovn-fips
  • /test e2e-aws-ovn-image-registry
  • /test e2e-aws-ovn-serial
  • /test e2e-gcp-ovn
  • /test e2e-gcp-ovn-builds
  • /test e2e-gcp-ovn-image-ecosystem
  • /test e2e-gcp-ovn-upgrade
  • /test e2e-metal-ipi-ovn-ipv6
  • /test images
  • /test lint
  • /test unit
  • /test verify
  • /test verify-deps

The following commands are available to trigger optional jobs:

  • /test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
  • /test e2e-agnostic-ovn-cmd
  • /test e2e-aws
  • /test e2e-aws-csi
  • /test e2e-aws-disruptive
  • /test e2e-aws-etcd-certrotation
  • /test e2e-aws-etcd-recovery
  • /test e2e-aws-ovn
  • /test e2e-aws-ovn-cgroupsv2
  • /test e2e-aws-ovn-etcd-scaling
  • /test e2e-aws-ovn-ipsec-serial
  • /test e2e-aws-ovn-kube-apiserver-rollout
  • /test e2e-aws-ovn-kubevirt
  • /test e2e-aws-ovn-single-node
  • /test e2e-aws-ovn-single-node-serial
  • /test e2e-aws-ovn-single-node-techpreview
  • /test e2e-aws-ovn-single-node-techpreview-serial
  • /test e2e-aws-ovn-single-node-upgrade
  • /test e2e-aws-ovn-upgrade
  • /test e2e-aws-ovn-upi
  • /test e2e-aws-proxy
  • /test e2e-azure
  • /test e2e-azure-ovn-etcd-scaling
  • /test e2e-azure-ovn-upgrade
  • /test e2e-baremetalds-kubevirt
  • /test e2e-external-aws
  • /test e2e-external-aws-ccm
  • /test e2e-gcp-csi
  • /test e2e-gcp-disruptive
  • /test e2e-gcp-fips-serial
  • /test e2e-gcp-ovn-etcd-scaling
  • /test e2e-gcp-ovn-rt-upgrade
  • /test e2e-gcp-ovn-techpreview
  • /test e2e-gcp-ovn-techpreview-serial
  • /test e2e-metal-ipi-ovn
  • /test e2e-metal-ipi-ovn-dualstack
  • /test e2e-metal-ipi-ovn-dualstack-local-gateway
  • /test e2e-metal-ipi-ovn-kube-apiserver-rollout
  • /test e2e-metal-ipi-serial
  • /test e2e-metal-ipi-serial-ovn-ipv6
  • /test e2e-metal-ipi-virtualmedia
  • /test e2e-openstack-ovn
  • /test e2e-openstack-serial
  • /test e2e-vsphere
  • /test e2e-vsphere-ovn-dualstack-primaryv6
  • /test e2e-vsphere-ovn-etcd-scaling
  • /test okd-e2e-gcp
  • /test okd-scos-images

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd
  • pull-ci-openshift-origin-master-e2e-aws-csi
  • pull-ci-openshift-origin-master-e2e-aws-ovn-cgroupsv2
  • pull-ci-openshift-origin-master-e2e-aws-ovn-edge-zones
  • pull-ci-openshift-origin-master-e2e-aws-ovn-fips
  • pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-kube-apiserver-rollout
  • pull-ci-openshift-origin-master-e2e-aws-ovn-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade
  • pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade
  • pull-ci-openshift-origin-master-e2e-gcp-csi
  • pull-ci-openshift-origin-master-e2e-gcp-ovn
  • pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade
  • pull-ci-openshift-origin-master-e2e-gcp-ovn-upgrade
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout
  • pull-ci-openshift-origin-master-e2e-openstack-ovn
  • pull-ci-openshift-origin-master-images
  • pull-ci-openshift-origin-master-lint
  • pull-ci-openshift-origin-master-unit
  • pull-ci-openshift-origin-master-verify
  • pull-ci-openshift-origin-master-verify-deps

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Aug 27 '24 15:08 openshift-ci[bot]

@stbenjam: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

  • /test e2e-aws-jenkins
  • /test e2e-aws-ovn-edge-zones
  • /test e2e-aws-ovn-fips
  • /test e2e-aws-ovn-image-registry
  • /test e2e-aws-ovn-serial
  • /test e2e-gcp-ovn
  • /test e2e-gcp-ovn-builds
  • /test e2e-gcp-ovn-image-ecosystem
  • /test e2e-gcp-ovn-upgrade
  • /test e2e-metal-ipi-ovn-ipv6
  • /test images
  • /test lint
  • /test unit
  • /test verify
  • /test verify-deps

The following commands are available to trigger optional jobs:

  • /test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
  • /test e2e-agnostic-ovn-cmd
  • /test e2e-aws
  • /test e2e-aws-csi
  • /test e2e-aws-disruptive
  • /test e2e-aws-etcd-certrotation
  • /test e2e-aws-etcd-recovery
  • /test e2e-aws-ovn
  • /test e2e-aws-ovn-cgroupsv2
  • /test e2e-aws-ovn-etcd-scaling
  • /test e2e-aws-ovn-ipsec-serial
  • /test e2e-aws-ovn-kube-apiserver-rollout
  • /test e2e-aws-ovn-kubevirt
  • /test e2e-aws-ovn-single-node
  • /test e2e-aws-ovn-single-node-serial
  • /test e2e-aws-ovn-single-node-techpreview
  • /test e2e-aws-ovn-single-node-techpreview-serial
  • /test e2e-aws-ovn-single-node-upgrade
  • /test e2e-aws-ovn-upgrade
  • /test e2e-aws-ovn-upi
  • /test e2e-aws-proxy
  • /test e2e-azure
  • /test e2e-azure-ovn-etcd-scaling
  • /test e2e-azure-ovn-upgrade
  • /test e2e-baremetalds-kubevirt
  • /test e2e-external-aws
  • /test e2e-external-aws-ccm
  • /test e2e-gcp-csi
  • /test e2e-gcp-disruptive
  • /test e2e-gcp-fips-serial
  • /test e2e-gcp-ovn-etcd-scaling
  • /test e2e-gcp-ovn-rt-upgrade
  • /test e2e-gcp-ovn-techpreview
  • /test e2e-gcp-ovn-techpreview-serial
  • /test e2e-metal-ipi-ovn
  • /test e2e-metal-ipi-ovn-dualstack
  • /test e2e-metal-ipi-ovn-dualstack-local-gateway
  • /test e2e-metal-ipi-ovn-kube-apiserver-rollout
  • /test e2e-metal-ipi-serial
  • /test e2e-metal-ipi-serial-ovn-ipv6
  • /test e2e-metal-ipi-virtualmedia
  • /test e2e-openstack-ovn
  • /test e2e-openstack-serial
  • /test e2e-vsphere
  • /test e2e-vsphere-ovn-dualstack-primaryv6
  • /test e2e-vsphere-ovn-etcd-scaling
  • /test okd-e2e-gcp
  • /test okd-scos-images

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd
  • pull-ci-openshift-origin-master-e2e-aws-csi
  • pull-ci-openshift-origin-master-e2e-aws-ovn-cgroupsv2
  • pull-ci-openshift-origin-master-e2e-aws-ovn-edge-zones
  • pull-ci-openshift-origin-master-e2e-aws-ovn-fips
  • pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-kube-apiserver-rollout
  • pull-ci-openshift-origin-master-e2e-aws-ovn-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade
  • pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade
  • pull-ci-openshift-origin-master-e2e-gcp-csi
  • pull-ci-openshift-origin-master-e2e-gcp-ovn
  • pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade
  • pull-ci-openshift-origin-master-e2e-gcp-ovn-upgrade
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout
  • pull-ci-openshift-origin-master-e2e-openstack-ovn
  • pull-ci-openshift-origin-master-images
  • pull-ci-openshift-origin-master-lint
  • pull-ci-openshift-origin-master-unit
  • pull-ci-openshift-origin-master-verify
  • pull-ci-openshift-origin-master-verify-deps

In response to this:

/test e2e-metal-ovn-kube-apiserver-rollout

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Aug 27 '24 15:08 openshift-ci[bot]

/test e2e-metal-ipi-ovn-kube-apiserver-rollout

stbenjam avatar Aug 27 '24 15:08 stbenjam

/test e2e-metal-ipi-ovn-kube-apiserver-rollout

stbenjam avatar Aug 27 '24 15:08 stbenjam

@stbenjam: This pull request references Jira Issue OCPBUGS-39030, which is invalid:

  • expected the bug to target the "4.18.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

We are sometimes leaving this test with kube-apiserver operator "Progressing" -- this triggers a failure in gather-extra of the "operator conditions kube-apiserver" test, which expects operators to be stable when the job finishes.

Before we return from this test, make sure kube-apiserver is not progressing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Aug 27 '24 15:08 openshift-ci-robot

/jira refresh

stbenjam avatar Aug 27 '24 15:08 stbenjam

@stbenjam: This pull request references Jira Issue OCPBUGS-39030, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.18.0) matches configured target version for branch (4.18.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Aug 27 '24 15:08 openshift-ci-robot

/test e2e-metal-ipi-ovn-kube-apiserver-rollout

stbenjam avatar Aug 27 '24 15:08 stbenjam

/jira backport release-4.17

stbenjam avatar Aug 27 '24 15:08 stbenjam

@stbenjam: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges: /cherrypick release-4.17

In response to this:

/jira backport release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Aug 27 '24 15:08 openshift-ci-robot

@openshift-ci-robot: once the present PR merges, I will cherry-pick it on top of release-4.17 in a new PR and assign it to you.

In response to this:

@stbenjam: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges: /cherrypick release-4.17

In response to this:

/jira backport release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

/test e2e-metal-ipi-ovn-kube-apiserver-rollout

stbenjam avatar Aug 27 '24 15:08 stbenjam

/lgtm /hold unhold when you are ready

neisw avatar Aug 27 '24 18:08 neisw

/retest-required

stbenjam avatar Aug 27 '24 18:08 stbenjam

/lgtm

deads2k avatar Aug 27 '24 20:08 deads2k

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, neisw, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • ~~OWNERS~~ [deads2k,neisw,stbenjam]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci[bot] avatar Aug 27 '24 20:08 openshift-ci[bot]

/hold cancel

This does what I wanted

stbenjam avatar Aug 28 '24 11:08 stbenjam

/retest-required

Remaining retests: 0 against base HEAD c53592a77542ceed003566ed1dc3de58a6b04e45 and 2 for PR HEAD d3d683d451bd9a7b95991f732b587f8a5c05ff57 in total

openshift-ci-robot avatar Aug 28 '24 12:08 openshift-ci-robot

Job Failure Risk Analysis for sha: d3d683d451bd9a7b95991f732b587f8a5c05ff57

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn Medium
[sig-network] pods should successfully create sandboxes by adding pod to network
This test has passed 96.15% of 26 runs on jobs ['periodic-ci-openshift-release-master-ci-4.18-e2e-gcp-ovn'] in the last 14 days.

Open Bugs
s390x: [sig-network] pods should successfully create sandboxes by adding pod to network fails with error adding pod to CNI network

openshift-trt-bot avatar Aug 28 '24 22:08 openshift-trt-bot

/retest-required

Remaining retests: 0 against base HEAD 4b28768eeed774acf2f073cd6574d291e695514c and 2 for PR HEAD d3d683d451bd9a7b95991f732b587f8a5c05ff57 in total

openshift-ci-robot avatar Aug 28 '24 22:08 openshift-ci-robot

@stbenjam: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-ipsec-serial d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-ipsec-serial
ci/prow/e2e-aws-ovn-upgrade d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-upgrade
ci/prow/e2e-aws-ovn-single-node d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-single-node
ci/prow/e2e-agnostic-ovn-cmd d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-agnostic-ovn-cmd
ci/prow/e2e-aws-ovn-single-node-serial d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-single-node-serial
ci/prow/e2e-aws-ovn-cgroupsv2 d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-cgroupsv2
ci/prow/e2e-openstack-ovn d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-openstack-ovn
ci/prow/e2e-aws-csi d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-csi
ci/prow/e2e-aws-ovn-kube-apiserver-rollout d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-kube-apiserver-rollout
ci/prow/e2e-aws-ovn-single-node-upgrade d3d683d451bd9a7b95991f732b587f8a5c05ff57 link false /test e2e-aws-ovn-single-node-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci[bot] avatar Aug 29 '24 02:08 openshift-ci[bot]

/retest-required

Remaining retests: 0 against base HEAD 4b28768eeed774acf2f073cd6574d291e695514c and 2 for PR HEAD d3d683d451bd9a7b95991f732b587f8a5c05ff57 in total

openshift-ci-robot avatar Aug 29 '24 02:08 openshift-ci-robot

@stbenjam: Jira Issue OCPBUGS-39030: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-39030 has been moved to the MODIFIED state.

In response to this:

We are sometimes leaving this test with kube-apiserver operator "Progressing" -- this triggers a failure in gather-extra of the "operator conditions kube-apiserver" test, which expects operators to be stable when the job finishes.

Before we return from this test, make sure kube-apiserver is not progressing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Aug 29 '24 15:08 openshift-ci-robot

@openshift-ci-robot: new pull request created: #29047

In response to this:

@stbenjam: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges: /cherrypick release-4.17

In response to this:

/jira backport release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.