origin icon indicating copy to clipboard operation
origin copied to clipboard

OCPBUGS-48186: Add kubelet and CRI-O panic detection invariant test

Open PannagaRao opened this issue 3 months ago • 68 comments

Signed-off-by: Pannaga Rao Bhoja Ramamanohara

PannagaRao avatar Sep 11 '25 20:09 PannagaRao

Risk analysis has seen new tests most likely introduced by this PR. Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 7570b8a1da7fb01bc67e401ca83780a3caef87db

  • "[sig-node] kubelet-log-collector detects kubelet or CRI-O panics" [Total: 38, Pass: 38, Fail: 0, Flake: 0]

openshift-trt[bot] avatar Sep 12 '25 01:09 openshift-trt[bot]

Job Failure Risk Analysis for sha: e2aae7e7ecadf59a578a3f17019fc1e8a315e03f

Job Name Failure Risk
pull-ci-openshift-origin-main-e2e-aws-disruptive High
operator conditions image-registry
This test has passed 99.44% of 3956 runs on release 4.21 [Overall] in the last week.

Open Bugs
Define New Post Analysis Command
pull-ci-openshift-origin-main-e2e-aws-ovn-single-node-upgrade IncompleteTests
Tests for this run (22) are below the historical average (4535): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

Risk analysis has seen new tests most likely introduced by this PR. Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: e2aae7e7ecadf59a578a3f17019fc1e8a315e03f

  • "[Monitor:kubelet-log-collector][sig-node] kubelet-log-collector detects kubelet or CRI-O panics" [Total: 11, Pass: 11, Fail: 0, Flake: 0]

openshift-trt[bot] avatar Sep 16 '25 00:09 openshift-trt[bot]

/test-with openshift/kubernetes#2468

PannagaRao avatar Sep 17 '25 20:09 PannagaRao

/test e2e-aws-ovn-fips

PannagaRao avatar Sep 17 '25 20:09 PannagaRao

/test e2e-aws-ovn-fips --from openshift/kubernetes#2468

PannagaRao avatar Sep 18 '25 00:09 PannagaRao

/test e2e-aws-ovn-fips --from=openshift/kubernetes#2468

PannagaRao avatar Sep 18 '25 00:09 PannagaRao

Job Failure Risk Analysis for sha: 973a936af679a6a624249f7fd89bff0f2303ccd7

Job Name Failure Risk
pull-ci-openshift-origin-main-e2e-aws-disruptive High
operator conditions monitoring
This test has passed 99.52% of 4175 runs on release 4.21 [Overall] in the last week.

Open Bugs
Define New Post Analysis Command
---
operator conditions image-registry
This test has passed 99.54% of 4174 runs on release 4.21 [Overall] in the last week.

Open Bugs
Define New Post Analysis Command
---
operator conditions ingress
This test has passed 99.57% of 4175 runs on release 4.21 [Overall] in the last week.

Open Bugs
Define New Post Analysis Command
pull-ci-openshift-origin-main-e2e-openstack-ovn IncompleteTests
Tests for this run (104) are below the historical average (1641): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

Risk analysis has seen new tests most likely introduced by this PR. Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 973a936af679a6a624249f7fd89bff0f2303ccd7

  • "[Monitor:kubelet-log-collector][sig-node] kubelet-log-collector detects kubelet or CRI-O panics" [Total: 36, Pass: 36, Fail: 0, Flake: 0]

openshift-trt[bot] avatar Sep 18 '25 04:09 openshift-trt[bot]

/payload-job-with-prs e2e-aws-ovn-fips openshift/kubernetes#2468

PannagaRao avatar Sep 18 '25 16:09 PannagaRao

@PannagaRao: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • e2e-aws-ovn-fips

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0915cc20-94aa-11f0-86a6-18150b14e243-0

openshift-ci[bot] avatar Sep 18 '25 16:09 openshift-ci[bot]

/payload-job-with-prs e2e-aws-ovn-fips https://github.com/openshift/kubernetes/pull/2468

PannagaRao avatar Sep 18 '25 20:09 PannagaRao

@PannagaRao: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

openshift-ci[bot] avatar Sep 18 '25 20:09 openshift-ci[bot]

/payload-job-with-prs e2e-aws-ovn-fips openshift/kubernetes#2468

PannagaRao avatar Sep 18 '25 20:09 PannagaRao

@PannagaRao: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • e2e-aws-ovn-fips

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/148ed9e0-94d0-11f0-9cf6-f13decee80c1-0

openshift-ci[bot] avatar Sep 18 '25 20:09 openshift-ci[bot]

Confirmed that the panic gets detected by simulating a kubelet panic for testing purposes https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/openshift-kubernetes-2468-openshift-origin-30243-nightly-4.12-e2e-aws-ovn-fips/1968777839868121088/artifacts/e2e-aws-ovn-fips/openshift-e2e-test/artifacts/junit/e2e-monitor-tests__20250918-231819.xml

PannagaRao avatar Sep 19 '25 17:09 PannagaRao

/payload-job-with-prs e2e-aws-ovn-fips https://github.com/openshift/kubernetes/pull/2468

PannagaRao avatar Sep 22 '25 06:09 PannagaRao

@PannagaRao: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

openshift-ci[bot] avatar Sep 22 '25 06:09 openshift-ci[bot]

/payload-job-with-prs e2e-aws-ovn-fips openshift/kubernetes#2468

PannagaRao avatar Sep 22 '25 06:09 PannagaRao

@PannagaRao: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • e2e-aws-ovn-fips

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/25f00e30-9781-11f0-9812-b580d71cf823-0

openshift-ci[bot] avatar Sep 22 '25 06:09 openshift-ci[bot]

/skip /retest-required /lgtm

haircommander avatar Sep 22 '25 16:09 haircommander

@dennisperiquet could you PTAL! Thanks!

PannagaRao avatar Sep 22 '25 18:09 PannagaRao

/approve

DennisPeriquet avatar Sep 23 '25 00:09 DennisPeriquet

/verified by @PannagaRao

PannagaRao avatar Sep 23 '25 01:09 PannagaRao

@PannagaRao: This PR has been marked as verified by @PannagaRao.

In response to this:

/verified by @PannagaRao

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Sep 23 '25 01:09 openshift-ci-robot

@PannagaRao: This pull request references Jira Issue OCPBUGS-48186, which is invalid:

  • expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Signed-off-by: Pannaga Rao Bhoja Ramamanohara

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Sep 23 '25 01:09 openshift-ci-robot

/jira refresh

PannagaRao avatar Sep 23 '25 18:09 PannagaRao

@PannagaRao: This pull request references Jira Issue OCPBUGS-48186, which is invalid:

  • expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Sep 23 '25 18:09 openshift-ci-robot

@PannagaRao: This pull request references Jira Issue OCPBUGS-57477, which is invalid:

  • expected the bug to target either version "4.21." or "openshift-4.21.", but it targets "4.20.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Signed-off-by: Pannaga Rao Bhoja Ramamanohara

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Sep 23 '25 18:09 openshift-ci-robot

@PannagaRao: This pull request references Jira Issue OCPBUGS-48186, which is invalid:

  • expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Signed-off-by: Pannaga Rao Bhoja Ramamanohara

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Sep 23 '25 18:09 openshift-ci-robot

/jira refresh

PannagaRao avatar Sep 23 '25 18:09 PannagaRao

@PannagaRao: This pull request references Jira Issue OCPBUGS-48186, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact: /cc @cpmeadors

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Sep 23 '25 18:09 openshift-ci-robot