origin icon indicating copy to clipboard operation
origin copied to clipboard

Automating OCP-55033 in Origin

Open asahay19 opened this issue 1 month ago • 18 comments

This case is about checking the Kubelet log level is 2. Here is the test case link : https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-55033

It is duplicate of PR: https://github.com/openshift/origin/pull/30460 PR: 30460 got closed because I had to delete repo, so creating new PR again with addressing all the comments.

PTAL @sairameshv @lyman9966 @cpmeadors

Here is the output for this PR . It got passed successfully while executing it on my local :

./openshift-tests run-test "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]"

Running Suite: - /Users/asahay/newOCP-55033/origin

Random Seed: 1764311478 - will randomize all specs

Will run 1 of 1 specs

[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL github.com/openshift/origin/test/extended/node/node_e2e/node.go:20 STEP: Creating a kubernetes client @ 11/28/25 12:01:22.453 I1128 12:01:22.455698 58826 discovery.go:214] Invalidating discovery information STEP: Polling to check kubelet log level on ready nodes @ 11/28/25 12:01:22.455 STEP: Getting all node names in the cluster @ 11/28/25 12:01:32.457 I1128 12:01:35.246189 58826 node.go:30] Node Names are ip-10-0-21-47.us-east-2.compute.internal ip-10-0-29-148.us-east-2.compute.internal ip-10-0-43-146.us-east-2.compute.internal ip-10-0-53-99.us-east-2.compute.internal ip-10-0-76-236.us-east-2.compute.internal ip-10-0-87-5.us-east-2.compute.internal STEP: Checking if node ip-10-0-21-47.us-east-2.compute.internal is Ready @ 11/28/25 12:01:35.246 I1128 12:01:36.338169 58826 node.go:37] Node ip-10-0-21-47.us-east-2.compute.internal Status is True

STEP: Checking KUBELET_LOG_LEVEL in kubelet.service on node ip-10-0-21-47.us-east-2.compute.internal @ 11/28/25 12:01:36.338
STEP: Checking kubelet process for --v=2 flag on node ip-10-0-21-47.us-east-2.compute.internal @ 11/28/25 12:01:44.469
STEP: Verifying KUBELET_LOG_LEVEL is set and kubelet is running with --v=2 @ 11/28/25 12:01:47.469

I1128 12:01:47.470106 58826 node.go:50] KUBELET_LOG_LEVEL is 2.

• [25.041 seconds]

Ran 1 of 1 Specs in 25.042 seconds SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped [ { "name": "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]", "lifecycle": "blocking", "duration": 25042, "startTime": "2025-11-28 06:31:22.430978 UTC", "endTime": "2025-11-28 06:31:47.473486 UTC", "result": "passed", "output": " STEP: Creating a kubernetes client @ 11/28/25 12:01:22.453\n STEP: Polling to check kubelet log level on ready nodes @ 11/28/25 12:01:22.455\n STEP: Getting all node names in the cluster @ 11/28/25 12:01:32.457\nI1128 12:01:35.246189 58826 node.go:30] \nNode Names are ip-10-0-21-47.us-east-2.compute.internal ip-10-0-29-148.us-east-2.compute.internal ip-10-0-43-146.us-east-2.compute.internal ip-10-0-53-99.us-east-2.compute.internal ip-10-0-76-236.us-east-2.compute.internal ip-10-0-87-5.us-east-2.compute.internal\n STEP: Checking if node ip-10-0-21-47.us-east-2.compute.internal is Ready @ 11/28/25 12:01:35.246\nI1128 12:01:36.338169 58826 node.go:37] \nNode ip-10-0-21-47.us-east-2.compute.internal Status is True\n\n STEP: Checking KUBELET_LOG_LEVEL in kubelet.service on node ip-10-0-21-47.us-east-2.compute.internal @ 11/28/25 12:01:36.338\n STEP: Checking kubelet process for --v=2 flag on node ip-10-0-21-47.us-east-2.compute.internal @ 11/28/25 12:01:44.469\n STEP: Verifying KUBELET_LOG_LEVEL is set and kubelet is running with --v=2 @ 11/28/25 12:01:47.469\nI1128 12:01:47.470106 58826 node.go:50] KUBELET_LOG_LEVEL is 2.\n\n" } ]%

asahay19 avatar Nov 21 '25 09:11 asahay19

Pipeline controller notification This repository is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci-robot avatar Nov 21 '25 09:11 openshift-ci-robot

Scheduling required tests: /test e2e-aws-csi /test e2e-aws-ovn-fips /test e2e-aws-ovn-microshift /test e2e-aws-ovn-microshift-serial /test e2e-aws-ovn-serial-1of2 /test e2e-aws-ovn-serial-2of2 /test e2e-gcp-csi /test e2e-gcp-ovn /test e2e-gcp-ovn-upgrade /test e2e-vsphere-ovn /test e2e-vsphere-ovn-upi /test e2e-aws-csi /test e2e-aws-ovn-fips /test e2e-aws-ovn-microshift /test e2e-aws-ovn-microshift-serial /test e2e-aws-ovn-serial-1of2 /test e2e-aws-ovn-serial-2of2 /test e2e-gcp-csi /test e2e-gcp-ovn /test e2e-gcp-ovn-upgrade /test e2e-vsphere-ovn /test e2e-vsphere-ovn-upi /test e2e-aws-csi /test e2e-aws-ovn-fips /test e2e-aws-ovn-microshift /test e2e-aws-ovn-microshift-serial /test e2e-aws-ovn-serial-1of2 /test e2e-aws-ovn-serial-2of2 /test e2e-gcp-csi /test e2e-gcp-ovn /test e2e-gcp-ovn-upgrade /test e2e-vsphere-ovn /test e2e-vsphere-ovn-upi

openshift-ci-robot avatar Nov 25 '25 10:11 openshift-ci-robot

/test e2e-vsphere-ovn-upi

asahay19 avatar Nov 25 '25 13:11 asahay19

/test e2e-vsphere-ovn

asahay19 avatar Nov 25 '25 13:11 asahay19

/lgtm

cpmeadors avatar Nov 25 '25 21:11 cpmeadors

/lgtm

lyman9966 avatar Nov 26 '25 09:11 lyman9966

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: asahay19, cpmeadors, lyman9966 Once this PR has been reviewed and has the lgtm label, please assign neisw for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci[bot] avatar Nov 26 '25 09:11 openshift-ci[bot]

I see where your test ran from the previous pr prior to 2989d03201

But I don't see that it ran in any presubmits on this pr.

neisw avatar Nov 27 '25 16:11 neisw

New changes are detected. LGTM label has been removed.

openshift-ci[bot] avatar Nov 28 '25 06:11 openshift-ci[bot]

I see where your test ran from the previous pr prior to 2989d03201

But I don't see that it ran in any presubmits on this pr.

yes, I have updated the PR description with the latest test output result. Thank you

asahay19 avatar Nov 28 '25 06:11 asahay19

Scheduling required tests: /test e2e-aws-csi /test e2e-aws-ovn-fips /test e2e-aws-ovn-microshift /test e2e-aws-ovn-microshift-serial /test e2e-aws-ovn-serial-1of2 /test e2e-aws-ovn-serial-2of2 /test e2e-gcp-csi /test e2e-gcp-ovn /test e2e-gcp-ovn-upgrade /test e2e-metal-ipi-ovn-ipv6 /test e2e-vsphere-ovn /test e2e-vsphere-ovn-upi

openshift-ci-robot avatar Nov 28 '25 07:11 openshift-ci-robot

Thanks, need to see what the issue is with those failures:

[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" [Total: 5, Pass: 0, Fail: 5, Flake: 0]

neisw avatar Nov 29 '25 21:11 neisw

/retest

asahay19 avatar Dec 01 '25 07:12 asahay19

/retest

asahay19 avatar Dec 02 '25 05:12 asahay19

/retest

asahay19 avatar Dec 09 '25 07:12 asahay19

Risk analysis has seen new tests most likely introduced by this PR. Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 94022ca737d4d2bd335279437e0cb8409dd3d4d9

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-fips High - "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that failed 4 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift High - "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that failed 4 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-gcp-ovn High - "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that was not present in all runs against the current commit, and also failed 3 time(s).
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6 High - "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that failed 4 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-vsphere-ovn High - "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that failed 4 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-vsphere-ovn-upi High - "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that was not present in all runs against the current commit, and also failed 3 time(s).

New tests seen in this PR at sha: 94022ca737d4d2bd335279437e0cb8409dd3d4d9

  • "[sig-node] Kubelet, CRI-O, CPU manager validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" [Total: 24, Pass: 2, Fail: 22, Flake: 0]

openshift-trt[bot] avatar Dec 09 '25 12:12 openshift-trt[bot]

I don't think this is a retest issue. Even if it was, a new known flaky test won't get approved.

neisw avatar Dec 09 '25 12:12 neisw

Scheduling required tests: /test e2e-aws-csi /test e2e-aws-ovn-fips /test e2e-aws-ovn-microshift /test e2e-aws-ovn-microshift-serial /test e2e-aws-ovn-serial-1of2 /test e2e-aws-ovn-serial-2of2 /test e2e-gcp-csi /test e2e-gcp-ovn /test e2e-gcp-ovn-upgrade /test e2e-metal-ipi-ovn-ipv6 /test e2e-vsphere-ovn /test e2e-vsphere-ovn-upi

openshift-ci-robot avatar Dec 19 '25 08:12 openshift-ci-robot

@asahay19: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-microshift 48664c1f21e2adf6c5cbd1a81d0e0fd1e1569afd link true /test e2e-aws-ovn-microshift
ci/prow/e2e-aws-ovn-serial-1of2 48664c1f21e2adf6c5cbd1a81d0e0fd1e1569afd link true /test e2e-aws-ovn-serial-1of2

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci[bot] avatar Dec 19 '25 12:12 openshift-ci[bot]

Risk analysis has seen new tests most likely introduced by this PR. Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 48664c1f21e2adf6c5cbd1a81d0e0fd1e1569afd

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift High - "[sig-node] [Jira:Node] Kubelet, CRI-O, CPU manager [OTP] validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit

New tests seen in this PR at sha: 48664c1f21e2adf6c5cbd1a81d0e0fd1e1569afd

  • "[sig-node] [Jira:Node] Kubelet, CRI-O, CPU manager [OTP] validate KUBELET_LOG_LEVEL [Suite:openshift/conformance/parallel]" [Total: 6, Pass: 5, Fail: 1, Flake: 0]

openshift-trt[bot] avatar Dec 19 '25 13:12 openshift-trt[bot]

A couple of comments:

  • Your annotation should be [Jira:"Node / Kubelet"]. Components should be valid Jira components and also match CR dashboard.
  • Test is failing the microshift job. You need to skip the test in microshift if it is not relevant there.

xueqzhan avatar Dec 22 '25 16:12 xueqzhan