argo-cd icon indicating copy to clipboard operation
argo-cd copied to clipboard

in-cluster isn't detected by ApplicationSet

Open kbreit opened this issue 7 months ago • 6 comments

Describe the bug

I created a simple ApplicationSet based on the ArgoCD documentation. It has a single cluster generator which should match all clusters. The ApplicationSet is never created because it isn't finding the in-cluster cluster and thus some values are not valid.

To Reproduce

Deploy ArgoCD with these Helm values.

global:
  domain: argocd.home.kevinbreit.net
  logging:
    level: debug
configs:
  params:
    server.insecure: true
server:
  service:
    type: ClusterIP
  ingress:
    enabled: true

Also, be sure to include the appset in the extraObjects section.

  - apiVersion: argoproj.io/v1alpha1
    kind: ApplicationSet
    metadata:
      name: infra
      namespace: argocd
    spec:
      goTemplate: true
      goTemplateOptions: ["missingkey=error"]
      generators:
        - clusters: {}
      template:
        metadata:
          name: "{{.server}}-infra"
          annotations:
            name: "{{.server}}-infra"
        spec:
          project: "default"
          source:
            repoURL: https://github.com/kbreit/k8s-infra/
            targetRevision: HEAD
            path: infra/apps
          destination:
            server: "https://kubernetes.default.svc"
            namespace: argocd

Verify the cluster is in ArgoCD

{"level":"warning","msg":"Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.","time":"2025-05-26T08:47:36-05:00"}
SERVER                          NAME        VERSION  STATUS   MESSAGE                                                  PROJECT
https://kubernetes.default.svc  in-cluster           Unknown  Cluster has no applications and is not being monitored.

Make sure the cluster secret exists with the proper label.

❯ kubectl describe secret -n argocd -l argocd.argoproj.io/secret-type=cluster
Name:         cluster-kubernetes.default.svc-3396314289
Namespace:    argocd
Labels:       argocd.argoproj.io/secret-type=cluster
Annotations:  argocd.argoproj.io/refresh: 2025-05-25T08:53:41Z
              managed-by: argocd.argoproj.io

Type:  Opaque

Data
====
config:  38 bytes
name:    10 bytes
server:  30 bytes

Check status of appset.

❯ argocd appset list
{"level":"warning","msg":"Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.","time":"2025-05-26T08:48:26-05:00"}
NAME          PROJECT  SYNCPOLICY  CONDITIONS                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      REPO                                  PATH        TARGET
argocd/infra  default  nil         [{ErrorOccurred Application.argoproj.io "-infra" is invalid: metadata.name: Invalid value: "-infra": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*') 2025-05-25 22:53:28 -0500 CDT True UpdateApplicationError} {ParametersGenerated Successfully generated parameters for all Applications 2025-05-25 22:53:28 -0500 CDT True ParametersGenerated}]  https://github.com/kbreit/k8s-infra/  infra/apps  HEAD

Expected behavior

I expected the appset to find the cluster and be created.

Version

argocd: v3.0.3+a14b012
  BuildDate: 2025-05-21T19:57:07Z
  GitCommit: a14b0125fe02ff953caffc59e55016e5872d45bf
  GitTreeState: clean
  GoVersion: go1.24.3
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v3.0.3+a14b012

Logs

time="2025-05-26T13:42:06Z" level=debug msg="clusters matching labels: 1" applicationset=infra namespace=argocd
time="2025-05-26T13:42:06Z" level=debug msg="Using flat mode = false for cluster generator" applicationset=infra namespace=argocd
time="2025-05-26T13:42:06Z" level=debug msg="matched cluster secret" applicationset=infra cluster=cluster-kubernetes.default.svc-3396314289 namespace=argocd
time="2025-05-26T13:42:06Z" level=debug msg="apps from generator: [{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:-infra GenerateName: Namespace:argocd SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[name:-infra] OwnerReferences:[] Finalizers:[resources-finalizer.argocd.argoproj.io] ManagedFields:[]} Spec:{Source:&ApplicationSource{RepoURL:https://github.com/kbreit/k8s-infra/,Path:infra/apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,Name:,} Destination:{Server:https://kubernetes.default.svc Namespace:argocd Name:} Project:default SyncPolicy:nil IgnoreDifferences:[] Info:[] RevisionHistoryLimit:<nil> Sources:[] SourceHydrator:nil} Status:{Resources:[] Sync:{Status: ComparedTo:{Source:{RepoURL: Path: TargetRevision: Helm:nil Kustomize:nil Directory:nil Plugin:nil Chart: Ref: Name:} Destination:{Server: Namespace: Name:} Sources:[] IgnoreDifferences:[]} Revision: Revisions:[]} Health:{Status: Message: LastTransitionTime:<nil>} History:[] Conditions:[] ReconciledAt:<nil> OperationState:nil ObservedAt:<nil> SourceType: Summary:{ExternalURLs:[] Images:[]} ResourceHealthSource: SourceTypes:[] ControllerNamespace: SourceHydrator:{LastSuccessfulOperation:nil CurrentOperation:nil}} Operation:nil}]" applicationset=argocd/infra generator="{nil &ClusterGenerator{Selector:{map[] []},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},SourceHydrator:nil,},},Values:map[string]string{},FlatList:false,} nil nil nil nil nil nil nil nil}"
time="2025-05-26T13:42:06Z" level=error msg="failed to unchanged Application" action=unchanged app=argocd/-infra applicationset=argocd/infra error="Application.argoproj.io \"-infra\" is invalid: metadata.name: Invalid value: \"-infra\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')"
time="2025-05-26T13:42:06Z" level=error msg="Reconciler error" ApplicationSet="{infra argocd}" controller=applicationset controllerGroup=argoproj.io controllerKind=ApplicationSet error="Application.argoproj.io \"-infra\" is invalid: metadata.name: Invalid value: \"-infra\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')" name=infra namespace=argocd reconcileID=9b90e3db-c3c8-40f3-b19a-8945d6f99165

kbreit avatar May 26 '25 13:05 kbreit

Looks like the problem is that {{.server}} contains characters that are not allowed in the fields in which it is used.

I'd recommend trying {{.name}}.

crenshaw-dev avatar May 27 '25 19:05 crenshaw-dev

@crenshaw-dev I'm getting the same error. I don't think it's finding either .server or .name so the name resolves to -infra which isn't legal.

k8s-infra on  main [✘!?] took 11s
❯ argocd appset list
{"level":"warning","msg":"Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.","time":"2025-05-27T16:32:15-05:00"}
NAME          PROJECT  SYNCPOLICY  CONDITIONS                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      REPO                                  PATH        TARGET
argocd/infra  default  nil         [{ErrorOccurred Application.argoproj.io "-infra" is invalid: metadata.name: Invalid value: "-infra": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*') 2025-05-25 22:53:28 -0500 CDT True UpdateApplicationError} {ParametersGenerated Successfully generated parameters for all Applications 2025-05-25 22:53:28 -0500 CDT True ParametersGenerated}]  https://github.com/kbreit/k8s-infra/  infra/apps  HEAD

k8s-infra on  main [✘!?]
❯ kubectl -n argocd get appset infra -oyaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2025-05-26T03:53:27Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: infra
  namespace: argocd
  resourceVersion: "67922818"
  uid: 9d03f662-43dc-4cb1-bbad-527df1c6b960
spec:
  generators:
  - clusters: {}
  goTemplate: true
  goTemplateOptions:
  - missingkey=error
  template:
    metadata:
      annotations:
        name: -infra
      name: -infra
    spec:
      destination:
        namespace: argocd
        server: https://kubernetes.default.svc
      project: default
      source:
        path: infra/apps
        repoURL: https://github.com/kbreit/k8s-infra/
        targetRevision: HEAD
status:
  conditions:
  - lastTransitionTime: "2025-05-26T03:53:28Z"
    message: 'Application.argoproj.io "-infra" is invalid: metadata.name: Invalid
      value: "-infra": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric
      characters, ''-'' or ''.'', and must start and end with an alphanumeric character
      (e.g. ''example.com'', regex used for validation is ''[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'')'
    reason: UpdateApplicationError
    status: "True"
    type: ErrorOccurred
  - lastTransitionTime: "2025-05-26T03:53:28Z"
    message: Successfully generated parameters for all Applications
    reason: ParametersGenerated
    status: "True"
    type: ParametersGenerated
  - lastTransitionTime: "2025-05-26T03:53:28Z"
    message: 'Application.argoproj.io "-infra" is invalid: metadata.name: Invalid
      value: "-infra": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric
      characters, ''-'' or ''.'', and must start and end with an alphanumeric character
      (e.g. ''example.com'', regex used for validation is ''[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'')'
    reason: UpdateApplicationError
    status: "False"
    type: ResourcesUpToDate

And here's my manifest.

  - apiVersion: argoproj.io/v1alpha1
    kind: ApplicationSet
    metadata:
      name: infra
      namespace: argocd
    spec:
      goTemplate: true
      goTemplateOptions: ["missingkey=error"]
      generators:
        - clusters: {}
      template:
        metadata:
          name: "{{.name}}-infra"
          annotations:
            name: "{{.name}}-infra"
        spec:
          project: "default"
          source:
            repoURL: https://github.com/kbreit/k8s-infra/
            targetRevision: HEAD
            path: infra/apps
          destination:
            server: "https://kubernetes.default.svc"
            namespace: argocd

kbreit avatar May 27 '25 21:05 kbreit

Oh, Helm is picking up the {{}} fields and replacing them with empty strings.

Try escaping the inner template.

name: "{{`{{.name}}`}}-infra"

crenshaw-dev avatar May 27 '25 21:05 crenshaw-dev

@crenshaw-dev Ahh good call thank you for finding that!

Do you think there's a place in the documentation to call this out? Most likely in the Helm chart documentation.

kbreit avatar May 28 '25 02:05 kbreit

Helm chart docs might be okay, but I'm not sure. extraObjects is generic, so adding docs that are specific to AppSets might not make a lot of sense.

crenshaw-dev avatar May 28 '25 02:05 crenshaw-dev

The only mention I could find in the doc is

ApplicationSet uses the same templating notation as Helm ({{}}). If the ApplicationSet templates aren't written as Helm string literals, Helm will throw an error like function "cluster" not defined. To avoid that error, write the template as a Helm string literal. For example:

metadata:
  name: '{{`{{ .nameNormalized }}`}}-guestbook'

This only applies if you use Helm to deploy your ApplicationSet resources.

Ref: https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/Template/#deploying-applicationset-resources-as-part-of-a-helm-chart

@crenshaw-dev Do you think the documentation could be improved to say somelike applicationset using helm will be rendered twice, first with helm using its own values and then by applicationset using values from generators and values intended for generators should be escaped so that they aren't swallowed by helm?

I would be happy to help and send a proper PR if it would be beneficial.

EricFortin avatar May 29 '25 10:05 EricFortin

If you see a way to rephrase the docs to be clearer, I'd be happy to review a PR! This is one of those things where you kinda just have to know the behavior of the tools you're working with... but the docs can hopefully alleviate a common point of confusion.

crenshaw-dev avatar Jun 03 '25 19:06 crenshaw-dev

@kbreit @crenshaw-dev I have made an attempt at improving the documentation. Let me know if it is better.

EricFortin avatar Jun 04 '25 11:06 EricFortin