argo-cd
argo-cd copied to clipboard
argoproj
namespace not found vs. CreateNamespace=true
Describe the bug
I'm getting a namespace not found error when deploying an app.
It worked yesterday but stopped working today.
Since yesterday all nodes have only been scaled down and up again. No configuration has been changed apart from the fact I deleted the app along with the namespace and was trying to deploy it again. I also tried with a different namespace but it didn't work.
App definition:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
finalizers:
- resources-finalizer.argocd.argoproj.io
labels:
argocd.argoproj.io/instance: test-hello-world-app
name: test-hello-world
namespace: test-hello-world
spec:
destination:
namespace: test-hello-world
server: 'https://kubernetes.default.svc'
project: sebolabs-test
source:
helm:
valueFiles:
- values.yaml
- values-test.yaml
path: apps/hello-world
repoURL: 'https://...'
targetRevision: trying-out-things
syncPolicy:
automated:
prune: true
retry:
backoff:
duration: 5s
factor: 2
maxDuration: 1m
limit: 1
syncOptions:
- CreateNamespace=true
The sebolabs-test project accepts test-* namespaces + ArgoCD is configured with application.namespaces: "test-*". Just like it was explained yesterday here.
As I was running out of ideas on how to get it working again I updated ArgoCD to v2.5.4 but that didn't change anything.
To Reproduce
See above.
Expected behavior
The test-hello-world namespace should be automatically created and the app should be successfully deployed.
Screenshots
N/A
Version
v2.5.4+86b2dde
Logs
time="2022-12-08T21:29:29Z" level=info msg=Syncing application=argocd/test-hello-world-app skipHooks=false started=false syncId=00008-QWOTG
time="2022-12-08T21:29:29Z" level=info msg="Tasks (dry-run)" application=argocd/test-hello-world-app syncId=00008-QWOTG tasks="[Sync/0 resource argoproj.io/Application:test-hello-world/test-hello-world nil->obj (,,)]"
time="2022-12-08T21:29:29Z" level=info msg="Applying resource Application/test-hello-world in cluster: https://172.20.0.1:443, namespace: test-hello-world"
time="2022-12-08T21:29:29Z" level=info msg="Updating operation state. phase: Running -> Running, message: 'one or more objects failed to apply, reason: namespaces \"test-hello-world\" not found. Retrying attempt #4 at 9:28PM.' -> 'one or more tasks are running'" application=argocd/test-hello-world-app syncId=00008-QWOTG
time="2022-12-08T21:29:29Z" level=info msg="Applying resource Application/test-hello-world in cluster: https://172.20.0.1:443, namespace: test-hello-world"
time="2022-12-08T21:29:29Z" level=info msg="Apply failed" application=argocd/test-hello-world-app dryRun=false message="namespaces \"test-hello-world\" not found" syncId=00008-QWOTG task="Sync/0 resource argoproj.io/Application:test-hello-world/test-hello-world nil->obj (,,)"
time="2022-12-08T21:29:29Z" level=info msg="Adding resource result, status: 'SyncFailed', phase: 'Failed', message: 'namespaces \"test-hello-world\" not found'" application=argocd/test-hello-world-app kind=Application name=test-hello-world namespace=test-hello-world phase=Sync syncId=00008-QWOTG
time="2022-12-08T21:29:29Z" level=info msg="Updating operation state. phase: Running -> Failed, message: 'one or more tasks are running' -> 'one or more objects failed to apply, reason: namespaces \"test-hello-world\" not found'" application=argocd/test-hello-world-app syncId=00008-QWOTG
.spec.syncPolicy.syncOption can be solved by adding PruneLast=true parameter @sebolabs
Unfortunately, that didn't work and I think it could only be considered (by design) if the app got created.
It keeps on saying that ... one or more objects failed to apply, reason: namespaces "test-hello-world" not found. Retrying attempt ...
How are you deploying the test-hello-world app? From an app-of-apps, applicationset, kubectl or other?
The error message appears to be nonsensical. For Argo CD to throw an error when trying to deploy an Application which is installed in test-hello-world, that Application must first exist. For the Application to exist in that namespace, the namespace must exist.
My theory is that you're deploying from an app-of-apps and that the parent app needs CreateNamespace=true. But that could be a bad guess.
Looking at the logs, I think argocd/test-hello-world-app is your parent app. That's the one that needs CreateNamespace=true.
That's correct, the argocd/test-hello-world-app is the parrent app and here's its manifest:
project: sebolabs-test
source:
repoURL: 'https://...'
path: hello-world
targetRevision: testing
destination:
server: 'https://kubernetes.default.svc'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- Validate=false
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- RespectIgnoreDifferences=true
retry:
limit: 10
backoff:
duration: 10s
factor: 2
maxDuration: 3m
$ kubectl get apps -A | grep hello-world
argocd test-hello-world-app OutOfSync Healthy
If you sync that app, it should create the namespace for the child app.
The thing is that I can't, the parent (top-level) app will be in sync if child items get synced. At least that's how I believe it works.
Here's what I see when I click Sync for the parent app:
the parent (top-level) app will be in sync if child items get synced
Isn't that okay? Why wouldn't you want the parent app to be in sync?
I think the order needs to go "sync parent app" (now child app exists, so you can...) "sync child app(s)".
There's currently no child app to sync.
I would, but you can see on the screenshot what's there to be synchronized when I hit Sync for the parent app:
The only resource is the child app (test-hello-world/test-hello-world), this is why I said I believe that the parent app can show up as in sync if the child app gets synced.
And the latter won't happen because the namespace is not there even though the create namespace sync option is enabled all over the place.
I don't get that because it worked and stopped after I deleted everything and tried to let it be recreated, just like I explained in the beginning.
See this, example:
I clicked Sync individually for both webhooks there and when they got synced the aws-load-balancer-controller app automatically appeared as Synced.
the latter won't happen because the namespace is not there even though the create namespace sync option is enabled all over the place
Gotcha. The parent app's sync CreateNamespace option should be all that matters. The order of syncing matters, because until the parent app is synced there is 1) probably no namespace called test-hello-world and 2) no child app to sync.
What happens if you try to sync the parent app? Do you get the error message that is in the original post?
Apologies for the back-and-forth. Understanding the app-of-apps setup was the part I was missing, even though I think all the info was in the original post. :-)
If the parent app is failing to sync despite the CreateNamespace field being set, that's a bug.
So again :) when I hit Sync for the parent app it is trying to create the child app and complains that the namespace is not there.
No worries. For testing, and as expected, I created the namespace with kubectl and everything got nicely synced. Then I went and deleted both the app and the namespace, redeployed the parent app, and ended up with the same error.
Okee-doke, here's a minimal reproduction of the issue:
app.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: test-hello-world-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/crenshaw-dev/argo-cd-11625
path: resource
destination:
server: 'https://kubernetes.default.svc'
# uncomment this, and the sync will work
#namespace: test-hello-world
syncPolicy:
syncOptions:
- CreateNamespace=true
contents of resource to be deployed:
apiVersion: v1
kind: ConfigMap
metadata:
name: test
namespace: test-hello-world
This conflicts with my understanding of how the CreateNamespace is supposed to work.
My belief was that it would create a namespace each time it encountered a resource destined for a namespace which does not exist.
The reality is that it just creates the namespace specified in spec.destination.namespace, regardless of whether any resource actually uses that namespace.
@ashutosh16, @leoluz was I just incorrect about how this feature works? Or is this a regression?
Issue was previously identified here: https://github.com/argoproj/argo-cd/issues/6997
My belief was that it would create a namespace each time it encountered a resource destined for a namespace which does not exist. The reality is that it just creates the namespace specified in spec.destination.namespace, regardless of whether any resource actually uses that namespace.@ashutosh16, @leoluz was I just incorrect about how this feature works? Or is this a regression?
That is correct. Clarified it in the docs to avoid this type of confusion in the future: https://github.com/argoproj/argo-cd/pull/11723
@sebolabs @crenshaw-dev can we close this issue as the behaviour is clarified?
@sebolabs if spec.destination.namespace doesn't solve your use case (i.e. you need to deploy to multiple or to not-yet-known namespaces), would you be willing to open an enhancement request?
tbh, I think creating the resources' namespaces should be the default behavior, but we'd need to add the feature on a new SyncOption to preserve backwards compatibility.
The ApplicationSet example of the git-generator-directory is a good example to reproduce this bug
Adding spec.destination.namespace does not change anything.
https://github.com/argoproj/argo-cd/tree/master/applicationset/examples/git-generator-directory
You just have to add the namespace restrictions to the project name used in the example. I could replicate it deploying this ApplicationSet from another Application. Not sure if it's the App of App that introduce this issue. I remember using this mechanism without any issue several month ago.
Nevermind, I forgot the obvious:
syncPolicy:
automated: {}
syncOptions:
- CreateNamespace=true
We are still seeing the same error with
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Also seeing it with the latest version of ArgoCD and K8s when targeting this kustomization.yml file with an ApplicationSet using the git generator with spec.template.spec.syncPolicy.syncOptions["CreateNamespace=true"] set.
# https://www.talos.dev/v1.5/kubernetes-guides/configuration/deploy-metrics-server/
resources:
- https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml
helmCharts:
- name: metrics-server
repo: https://kubernetes-sigs.github.io/metrics-server/
version: 3.11.0 # https://artifacthub.io/packages/helm/metrics-server/metrics-server?modal=changelog
releaseName: metrics-server
namespace: metrics-server
valuesFile: values.yml
includeCRDs: true
Just had this issue when trying to recreate an app. I had to remove the namespace to purge some annotations we set with managedNamespaceMetadata. We are running v2.6.14, though.
It's an app-of-apps structure where I removed the whole app from the mother-app but it was still unable to recreate the namespace. It could only recreate the cluster-scoped resources.
EDIT: Nevermind. And on a totally unrelated note: never use "Prettier - Code Formatter" for VSCode without double-checking the changes it has made...
Hi @sebolabs, given your original app definition above it can be that you try to deploy your argocd application into the wrong namespace (test-hello-world), because your metadata.namespace and your spec.destination.namespace are identical. Is your ArgoCD application really deployed into "test-hello-world" namespace? Otherwise you need to change the metadata.namespace to the namespace where you deployed ArgoCD itself.
