origin icon indicating copy to clipboard operation
origin copied to clipboard

Use kubeconfig from secret mount instead of /tmp

Open stbenjam opened this issue 3 years ago • 7 comments

Every now and then we get catastrophic runs where hundreds of tests fail with this error:

error: stat /tmp/kubeconfig-2448219175: no such file or directory

Example run at [1].

This /tmp/kubeconfig file is coming from initializeTestFramework, which is only called by ceratin suites including upgrades. It in turn calls a vendored kube function that writes this /tmp/kubeconfig, derived from the service account in /var/run[2].

It's not clear to me why it's using this kubeconfig, instead of the one already present in /var/run mounted by ci-operator. Regular conformance tests I believe use this kubeconfig.

The benefit to this is the secret is mounted read only, so it shouldn't be modifable or removable. I dug around and I cannot figure what exactly caused the /tmp/kubeconfig- to be deleted, but I am hoping this guards against it.

This also remove a meaningless call that's efectively doing os.Setenv(os.Getenv("KUBECONFIG")).

[1] https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-sdn-upgrade/1572280613627498496 [2] https://github.com/openshift/origin/blob/199ad942ed2d5998b83da2a609e0a11e70e5475a/vendor/k8s.io/kubernetes/test/e2e/framework/test_context.go#L453

stbenjam avatar Sep 21 '22 14:09 stbenjam

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

openshift-ci[bot] avatar Sep 21 '22 14:09 openshift-ci[bot]

/test

stbenjam avatar Sep 21 '22 14:09 stbenjam

/test e2e-gcp-ovn-upgrade /test e2e-aws-upgrade

stbenjam avatar Sep 21 '22 14:09 stbenjam

@stbenjam: The /test command needs one or more targets. The following commands are available to trigger required jobs:

  • /test e2e-aws-image-registry
  • /test e2e-aws-jenkins
  • /test e2e-aws-ovn-fips
  • /test e2e-aws-ovn-serial
  • /test e2e-gcp-builds
  • /test e2e-gcp-image-ecosystem
  • /test e2e-gcp-ovn
  • /test e2e-gcp-ovn-upgrade
  • /test extended_gssapi
  • /test extended_ldap_groups
  • /test extended_networking
  • /test images
  • /test lint
  • /test verify
  • /test verify-deps

The following commands are available to trigger optional jobs:

  • /test e2e-agnostic-ovn-cmd
  • /test e2e-aws
  • /test e2e-aws-csi
  • /test e2e-aws-csi-migration
  • /test e2e-aws-disruptive
  • /test e2e-aws-multitenant
  • /test e2e-aws-ovn
  • /test e2e-aws-ovn-cgroupsv2
  • /test e2e-aws-ovn-single-node
  • /test e2e-aws-ovn-single-node-serial
  • /test e2e-aws-ovn-single-node-upgrade
  • /test e2e-aws-proxy
  • /test e2e-aws-upgrade
  • /test e2e-azure
  • /test e2e-gcp-csi
  • /test e2e-gcp-disruptive
  • /test e2e-gcp-fips-serial
  • /test e2e-gcp-ovn-rt-upgrade
  • /test e2e-metal-ipi
  • /test e2e-metal-ipi-ovn-dualstack
  • /test e2e-metal-ipi-ovn-ipv6
  • /test e2e-metal-ipi-serial
  • /test e2e-metal-ipi-serial-ovn-ipv6
  • /test e2e-metal-ipi-virtualmedia
  • /test e2e-openstack
  • /test e2e-openstack-serial
  • /test e2e-vsphere
  • /test okd-e2e-gcp

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd
  • pull-ci-openshift-origin-master-e2e-aws-csi
  • pull-ci-openshift-origin-master-e2e-aws-ovn-cgroupsv2
  • pull-ci-openshift-origin-master-e2e-aws-ovn-fips
  • pull-ci-openshift-origin-master-e2e-aws-ovn-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial
  • pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade
  • pull-ci-openshift-origin-master-e2e-gcp-builds
  • pull-ci-openshift-origin-master-e2e-gcp-csi
  • pull-ci-openshift-origin-master-e2e-gcp-ovn
  • pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade
  • pull-ci-openshift-origin-master-e2e-gcp-ovn-upgrade
  • pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6
  • pull-ci-openshift-origin-master-images
  • pull-ci-openshift-origin-master-lint
  • pull-ci-openshift-origin-master-verify
  • pull-ci-openshift-origin-master-verify-deps

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci[bot] avatar Sep 21 '22 14:09 openshift-ci[bot]

/test e2e-azure /test e2e-gcp-csi

stbenjam avatar Sep 21 '22 14:09 stbenjam

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci[bot] avatar Sep 21 '22 14:09 openshift-ci[bot]

@stbenjam: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-csi 2073953875815fe64050e0a3d6e3904f1528d148 link false /test e2e-gcp-csi
ci/prow/e2e-aws-upgrade 2073953875815fe64050e0a3d6e3904f1528d148 link false /test e2e-aws-upgrade
ci/prow/e2e-gcp-ovn-upgrade 2073953875815fe64050e0a3d6e3904f1528d148 link true /test e2e-gcp-ovn-upgrade
ci/prow/e2e-azure 2073953875815fe64050e0a3d6e3904f1528d148 link false /test e2e-azure

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci[bot] avatar Sep 21 '22 16:09 openshift-ci[bot]