k8s-deploy icon indicating copy to clipboard operation
k8s-deploy copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Cannot apply resources with defined namespaces in manifest

Open jesusha123 opened this issue 3 years ago • 23 comments

Repro steps:

  • Obtain a manifest with defined namespaces (e.g. cert-manager, ingress-nginx)
    • https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml
    • https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml
  • Add k8s-deploy step that deploys to kubernetes and commit, do not specify a namespece

Current behaviour:

  • Kubernetes deployment fails
Error: the namespace from the provided object "kube-system" does not match the namespace "cert-manager". You must pass '--namespace=kube-system' 

Error: the namespace from the provided object "ingress-nginx" does not match the namespace "default". You must pass '--namespace=ingress-nginx'

Expected behaviour:

  • k8s-deploy should not pass namespace=default parameter to kubectl if namespace is not defined in github action

The reasoning is, official instructions for ingress-nginx, cert-manager and others do not need a namespace parameter in kubectl, and if namespace=default is passed, the command fails. I have to split the manifests into multiple manifests per each namespace as a workaround.

jesusha123 avatar Dec 20 '21 20:12 jesusha123

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar Jan 03 '22 21:01 github-actions[bot]

Hello! We will look into this. Thanks!

OliverMKing avatar Mar 03 '22 20:03 OliverMKing

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar Mar 17 '22 21:03 github-actions[bot]

i was able to repro, and just made a PR

davidgamero avatar Jun 14 '22 16:06 davidgamero

merged the fix. please reopen if issue persists

davidgamero avatar Jun 15 '22 14:06 davidgamero

@davidgamero I believe I'm encountering this issue with k8s-deploy@v4 but I cannot reopen this issue.

I'm trying to deploy cert-manager 1.10.1. cert-manager is normally installed using kubectl apply -f. I put the manifest from https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml in my repo and ran the following action:

      - name: Deploy cert manager (pre-requisite for actions-runner-controller)
        uses: Azure/k8s-deploy@v4
        with:
          action: deploy
          strategy: basic
          manifests: |
            manifests/deployments/cert-manager-v1.10.1.yml
          force: true

Log:

Run Azure/k8s-deploy@v4
  with:
    action: deploy
    strategy: basic
    manifests: manifests/deployments/cert-manager-v1.10.1.yml
  
    force: true
    namespace: default
    pull-images: true
    route-method: service
    version-switch-buffer: 0
    traffic-split-method: pod
    percentage: 0
    token: ***
    annotate-namespace: true
    private-cluster: false
    skip-tls-verify: false
  env:
    AZURE_HTTP_USER_AGENT: 
    AZUREPS_HOST_ENVIRONMENT: 
    KUBECONFIG: /home/runner/work/_temp/kubeconfig_16696[2](https://github.com/BNGBank/github-self-hosted-runners/actions/runs/3563241016/jobs/5985804619#step:9:2)622506[4](https://github.com/BNGBank/github-self-hosted-runners/actions/runs/3563241016/jobs/5985804619#step:9:4)
    KUBE_CONFIG_PATH: /home/runner/work/_temp/kubeconfig_166962622[5](https://github.com/BNGBank/github-self-hosted-runners/actions/runs/3563241016/jobs/5985804619#step:9:5)0[6](https://github.com/BNGBank/github-self-hosted-runners/actions/runs/3563241016/jobs/5985804619#step:9:6)4
Deploying manifests
  /opt/hostedtoolcache/kubectl/1.25.4/x64/kubectl apply -f /tmp/cert-manager-v1.10.1.yml --force --namespace default
  namespace/cert-manager unchanged
  customresourcedefinition.apiextensions.k[8](https://github.com/BNGBank/github-self-hosted-runners/actions/runs/3563241016/jobs/5985804619#step:9:8)s.io/clusterissuers.cert-manager.io configured
  customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
  customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
  customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured
  customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
  customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-view configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-edit configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests configured
  clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests configured
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews configured
  mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
  validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  Error: Error: undefined

hansmbakker avatar Nov 28 '22 10:11 hansmbakker

I put the manifest from https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml in my repo and ran the following action:

Try it with the namespace: cert-manager option.

OliverMKing avatar Nov 28 '22 14:11 OliverMKing

Try it with the namespace: cert-manager option.

I guess that is no solution because the manifest does not only reference the cert-manager namespace but also the kube-system namespace, and with your suggestion it is only possible to use 1 namespace?

hansmbakker avatar Nov 28 '22 14:11 hansmbakker

As a workaround for now, I'd suggest using multiple k8s-deploy commands in your YAML like so:

      - name: Deploy to first namespace
        uses: Azure/k8s-deploy@v4
        with:
          action: deploy
          strategy: basic
          namespace: (FIRST_NAMESPACE)
          manifests: |
            path/to/manifests/for/first/ns
          force: true
      - name: Deploy to second namespace
        uses: Azure/k8s-deploy@v4
        with:
          action: deploy
          strategy: basic
          namespace: (SECOND_NAMESPACE)
          manifests: |
            path/to/manifests/for/second/ns
          force: true

We recently patched an issue where kubectl errors were failing silently, causing the action to pass even if a kubectl command necessary to execute the action had failed. As a result, users were seeing rollout failures even though their resources had successfully deployed because our action was checking the default namespace (if no namespace was provided, but if one was provided to the action it would check that one) for the deployed resources, but since the resources would be deployed to the namespace specified in the YAML, the rollout check would not be able to find the deployed resource, thereby causing the action to fail.

Our solution was to make sure that if no namespace was provided to the action, kubectl would deploy and perform a rollout check on the default namespace, or to do the same to whatever other namespace may be provided. Of course, this leads to the issue that you ran into, where the namespace being used by the action in the kubectl command it runs (default) is different from the namespace specified in your deployment YAML, and kubectl fails as a result.

We plan on working on a fix to support deployment to multiple namespaces within a single run by checking if deployment manifests specify a namespace, but I recommend using the above workaround until we release a new version with this functionality.

I hope this helps!

jaiveerk avatar Nov 28 '22 16:11 jaiveerk

That suggested workaround means I have to take apart the manifest that is provided by cert-manager.

I prefer to stick with a manual kubectl apply one this case as a workaround.

hansmbakker avatar Nov 28 '22 18:11 hansmbakker

@hansmbakker we will keep you updated on this. We are working on a fix.

OliverMKing avatar Nov 28 '22 19:11 OliverMKing

Thank you both and thank you @jaiveerk for the explanation!

hansmbakker avatar Nov 28 '22 19:11 hansmbakker

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar Dec 12 '22 21:12 github-actions[bot]

Not released yet but a fix is merged in

OliverMKing avatar Feb 06 '23 18:02 OliverMKing

Not released yet but a fix is merged in

Not sure if I should open a new issue but I tried using azure/k8s-deploy@main in order to test out the fix your merged, but I got this error

Error: File not found: '/home/myorg/_work/_actions/azure/k8s-deploy/main/lib/index.js'

Tchekda avatar Feb 14 '23 02:02 Tchekda

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar Feb 28 '23 03:02 github-actions[bot]

Hi @OliverMKing any ETA pls ? Issues was reported last year but issue is still in a place. I have same problem with prometheus stack.

vojtechvelkjop avatar May 17 '23 11:05 vojtechvelkjop

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar May 31 '23 12:05 github-actions[bot]

Not released yet but a fix is merged in

Not sure if I should open a new issue but I tried using azure/k8s-deploy@main in order to test out the fix your merged, but I got this error

Error: File not found: '/home/myorg/_work/_actions/azure/k8s-deploy/main/lib/index.js'

@Tchekda our release process doesn't store the final action on the main branch, and instead tags release branches, so unfortunately you can't test it out by referencing our main branch.

@vojtechvelkjop we are updating our release action due to a change in github action permissions which has blocked our normal release process, but we are fixing it now

davidgamero avatar May 31 '23 14:05 davidgamero

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar Jun 14 '23 18:06 github-actions[bot]

Tried latest 4.9 and 4.10 releases, but issue is still here:

the namespace from the provided object "monitoring" does not match the namespace "default". You must pass '--namespace=monitoring' to perform this operation

zigmund avatar Oct 13 '23 07:10 zigmund

This issue is idle because it has been open for 14 days with no activity.

github-actions[bot] avatar Oct 27 '23 12:10 github-actions[bot]

This issue I believe should have been fixed in version v4.1.0.0 by this https://github.com/Azure/k8s-deploy/pull/276

I've tried to use it but still getting the same error as others

...
...
Run azure/[email protected]
  with:
    manifests: kubernetes/cert-manager-v1.14.3-k8s-deployment.yaml
  
    namespace: default
    pull-images: true
    strategy: basic
    route-method: service
    version-switch-buffer: 0
    traffic-split-method: pod
    percentage: 0
    action: deploy
    force: false
    token: ***
    annotate-resources: true
    annotate-namespace: true
    private-cluster: false
    skip-tls-verify: false
  env:
    ...
Deploying manifests
  /usr/bin/kubectl apply -f /tmp/cert-manager-v1.14.[3](https://github.com/.......#step:6:3)-k[8](https://github.com/.......deployment.yaml --namespace default
  namespace/cert-manager unchanged
  customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io unchanged
  customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io unchanged
  customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io unchanged
  customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io unchanged
  customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io unchanged
  customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests unchanged
  clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests unchanged
  clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews configured
  mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
  validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "kube-system" does not match the namespace "default". You must pass '--namespace=kube-system' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  the namespace from the provided object "cert-manager" does not match the namespace "default". You must pass '--namespace=cert-manager' to perform this operation.
  Error: Error: undefined
...
...

motcke avatar Mar 06 '24 22:03 motcke