Ignoring Extraneous Resources doesn't improve app's overall sync status

[jfroche]

Describe the bug

We configure our openshift identity providers resources using argocd. Extra Secrets are created in the openshift-authentication namespace by the operator. We followed https://argoproj.github.io/argo-cd/user-guide/compare-options/#ignoring-resources-that-are-extraneous to ignore these secrets but our Application keeps being marked as "OutOfSync".

As you can see in the screenshot bellow the status the resource are correctly ignored but the application keeps being in OutOfSync.

To Reproduce

Here is our argocd application yaml:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: idp
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-options: Prune=false
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: cicd
  source:
    path: sbx/application/idp
    plugin:
      name: encrypted-yaml
    repoURL: http://<our internal repo>/projects.git
    targetRevision: cicd
  syncPolicy:
    automated:
      prune: false
      selfHeal: false

Here is our idp configuration:

---
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
    - name: local_accounts
      mappingMethod: claim
      type: HTPasswd
      htpasswd:
        fileData:
          name: htpass-users
    - name: Active_Directory
      mappingMethod: claim
      type: LDAP
      ldap:
        attributes:
          id:
            - dn
          email:
            - mail
          name:
            - cn
          preferredUsername:
            - sAMAccountName
        bindDN: "<redacted>"
        bindPassword:
          name: ad-secret
        insecure: true
        url: "ldap://<redacted>?sAMAccountName"

Expected behavior

The project should be marked as in sync.

Screenshots

Here is a screenshot of the project:

Here is a screenshot of the sync status:

Version

argocd: v1.6.1+159674e
  BuildDate: 2020-06-19T00:39:46Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64

Logs

Hitting refresh, here is the logs from the application controller:

time="2020-10-05T16:17:15Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=idp
time="2020-10-05T16:17:15Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=cicd-applications
time="2020-10-05T16:17:15Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=idp
time="2020-10-05T16:17:16Z" level=info msg="getRepoObjs stats" application=idp build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=77 unmarshal_ms=76 version_ms=0
time="2020-10-05T16:17:16Z" level=info msg="Skipping auto-sync: need to prune extra resources only but automated prune is disabled" application=idp
time="2020-10-05T16:17:16Z" level=info msg="Update successful" application=idp
time="2020-10-05T16:17:16Z" level=info msg="Reconciliation completed" application=idp dedup_ms=0 dest-namespace=default dest-server="https://kubernetes.default.svc" diff_ms=3 fields.level=2 git_ms=77 health_ms=0 live_ms=1 settings_ms=0 sync_ms=0 time_ms=97

Thank you for you help !

[jannfis]

Is there actually anything in the app diff?

[jfroche]

Yes the app diff contains the two extraneous secrets. Here is the application diff:

Here is the diff of one of the extraneous secret:

[jessesuen]

These annotations are supposed to be put on the resources you want to ignore and avoid pruning (i.e. the Secrets), not on the application spec.

  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-options: Prune=false

How did these resources end up with the Argo CD tracking label? Were they at one point deployed using Argo CD?

[jfroche]

The extra resources are created by the operator so we have no control on their content (they are recreated by openshift if we try to remove them). Is there a way to exclude them in argocd from the application as we have no control on their content ?

I will do maybe a PR to improve the documentation because This can be done by adding this annotation:in https://argoproj.github.io/argo-cd/user-guide/compare-options/#ignoring-resources-that-are-extraneous doesn't tell us where to add these annotations

[reggie-k]

Any news on the issue apart from changing the docs? Experiencing the same with OCP 4.5 oauth + console customizations

[reggie-k]

I think that for such resources, that are being created without user's control (kustomize is still a case of user control, but redhat operators aren't), the option should be available on the argo-app itself. Probably something like :

    syncPolicy:
      ignoreExtraneous:
         resourceNameRegex: <some regex matching the resource name>
         namespace: <the namespace name, as in those cases it differs from the targetNamespace of the app>

[reggie-k]

These annotations are supposed to be put on the resources you want to ignore and avoid pruning (i.e. the Secrets), not on the application spec.

  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-options: Prune=false

How did these resources end up with the Argo CD tracking label? Were they at one point deployed using Argo CD?

Argo app manages a user-created secret. Redhat operator copies that secret to another namespace and renames it, but copies the original secret along with the label that Argocd puts on it :-) So, from Argocd's point of view, the openshift-managed secret was created by it as well (though not true). I can raise an issue with redhat, I guess, as well, and ask to stop copying the labels from the original resource, but they might have a legit explanation for such a behavior. So, like I mentioned above, I think Argo app spec should have an option to exclude such resources

[frimik]

Yes, would also like an option to exclude a resource entirely. In this case they are resources created by helm from an application that defines a helm chart via the k3s HelmChart CRD ...

ignoreDifferences but without the jsonPointers basically ... or a functioning root (/) jsonPointer ... ?

  ignoreDifferences:
  - group: core
    kind: Secret
    name: concourse-postgresql
    jsonPointers:
    - /

[tsunamishaun]

Currently trying to get argocd to manage itself without using kustomize. I was hopeful that ignoreDifferences on the app would have allowed us to generate the configmap elsewhere along with maybe adding sync-options: Prune=false on the resource itself. Without implementing kustomize I don't see a way around this (but open to ideas?). I suspect it would be easier to implement an application specific ignoreDifferences than break out the argocd-cm cr into its formal parts (repositories cr, oidc cr etc) which would also be great.

[dkravetz]

I am running into a similar issue with Confluent's Operator for Kafka and related services.

Their operator creates some services and configmaps that Argo (understandably so) tries to prune, however, these resources are crucial for the functioning of our software.

Given that these resources are not part of the repo itself, is there any way to flag them so that Argo won't try to prune them? The approach of adding labels to each individual resources doesn't seem to apply here, given that a new service per replica is spun up

[jannfis]

I was wondering, do these operators actually set an ownerReferences on the resources they create?

[reggie-k]

At least openshift oauth cr does not have an owner reference On Wed, Mar 17, 2021, 13:28 jannfis @.***> wrote:

I was wondering, do these operators actually set an ownerReferences on the resources they create?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-cd/issues/4487#issuecomment-801007266, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVDWBGN5D3ALAX5ZB7UPH3TECG4HANCNFSM4SE5J6CA .

[dkravetz]

I was wondering, do these operators actually set an ownerReferences on the resources they create?

Not that I can see, but they do have the following labels:

  labels:
    argocd.argoproj.io/instance: schema-registry
    component: schemaregistry

[diranged]

We just ran into this same problem with the Strimzi.io Kafka Operator (slack thread https://argoproj.slack.com/archives/CASHNF6MS/p1616683667362100). There needs to be a way to have Argo completely ignore a resource and just not show it to us. :/

[dkravetz]

We just ran into this same problem with the Strimzi.io Kafka Operator (slack thread https://argoproj.slack.com/archives/CASHNF6MS/p1616683667362100). There needs to be a way to have Argo completely ignore a resource and just not show it to us. :/

I would personally prefer to see the resources, just not have the UI showing the need to prune this resources. After all, they are part of the Application, so I would assume that it's a better fitting use case to see the resources, just not have them affect the sync status

[rayman-g]

Same issue with victoriametrics operator, Argocd marks resources created by operator as "OutOfSync (requires pruning)" this are sa, cr, crb, psp

[timspeetjens]

Creating the original secrets with the IgnoreExtranous compare-option seems to be a valid workaround for the openshift-config related secrets:

The annotation is copied to the v4-0-config-user-idp-1-bind-password secret in theopenshift-authentication namespace, so the compare option is now effective on both itself and the declared secret (ad-secret in openshift-config). Unfortunately, this breaks purging of openshift-config secrets that are no longer used. Probably deleting it manually, is acceptable given that identity providers should not change too often.

---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    description: Bind password to access AD
  name: ad-secret
  namespace: openshift-config
type: Opaque
data:
  bindPassword: KioqKg==

[reggie-k]

Works for me

On Wed, May 12, 2021, 16:55 Tim Speetjens @.***> wrote:

Creating the original secrets with the IgnoreExtranous compare-option seems to be a valid workaround for the openshift-config related secrets:

The annotation is copied to the v4-0-config-user-idp-1-bind-password secret in theopenshift-authentication namespace, so the compare option is now effective on both itself and the declared secret (ad-secret in openshift-config). Unfortunately, this breaks purging of openshift-config secrets that are no longer used. Probably deleting it manually, is acceptable given that identity providers should not change too often.

---apiVersion: v1kind: Secretmetadata: annotations: argocd.argoproj.io/compare-options: IgnoreExtraneous description: Bind password to access AD name: ad-secret namespace: openshift-configtype: Opaquedata: bindPassword: KioqKg==

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-cd/issues/4487#issuecomment-839792534, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVDWBDCADMIMBRLNFXRLHTTNKCEJANCNFSM4SE5J6CA .

[timspeetjens]

RHBZ https://bugzilla.redhat.com/show_bug.cgi?id=1960651

[jannfis]

Possible solution to this: #6324

[gorantornqvist]

Hi, Any update?

"Possible solution to this: #6324"

I had trouble finding what the workaround is for making argocd not to track these extra resources created by openshift operators ...

Thanks ...

EDIT: So to clarify, the resources have argocd.argoproj.io/compare-options: IgnoreExtraneous and this fixes the SyncStatus issue. They are however still visible in "App Diff" which is confusing.

[gurnben]

I've also just encountered this issue when deploying the cert-manager operator on Argo 2.1.2.

[vikas027]

I think that for such resources, that are being created without user's control (kustomize is still a case of user control, but redhat operators aren't), the option should be available on the argo-app itself. Probably something like :

    syncPolicy:
      ignoreExtraneous:
         resourceNameRegex: <some regex matching the resource name>
         namespace: <the namespace name, as in those cases it differs from the targetNamespace of the app>

Hey @reggie-k , I can't find resourceNameRegex option in the code or the documentation. Are you sure this works?

[fadecore]

It was a proposal, not an implemented feature

[vikas027]

It was a proposal, not an implemented feature

Ah, my bad. I need new glasses :(

[aschaber1]

@jessesuen you removed the bug and added works-for-me label, I believe this is neither a bug nor a works-for-me situation, it's rather a feature or enhancement request. What do you think?

[jessesuen]

Since this issue was first created, we have a new annotation-based resource tracking mechanism that will help us identify resources that are offshoots of managed resources and allow us to decide not to prune them. Please follow this issue:

https://github.com/argoproj/argo-cd/issues/8683#issuecomment-1111605207

[reggie-k]

I think that for such resources, that are being created without user's control (kustomize is still a case of user control, but redhat operators aren't), the option should be available on the argo-app itself. Probably something like :

    syncPolicy:
      ignoreExtraneous:
         resourceNameRegex: <some regex matching the resource name>
         namespace: <the namespace name, as in those cases it differs from the targetNamespace of the app>

Hey @reggie-k , I can't find resourceNameRegex option in the code or the documentation. Are you sure this works?

As pointed out above, this was a proposal. But what does work for me, is setting the two annotations on the original resource I manage in git:

 annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-options: Prune=false

Then, the resource created by operator (the unmanaged one, the one that is created by the operator from the original one in git) becomes yellow (as requires pruning), but the application itself is green and synced.

[RSBlek]

Is there a way to hide the ignored resources in the interface as well?

The two annotations

      argocd.argoproj.io/compare-options: IgnoreExtraneous
      argocd.argoproj.io/sync-options: Prune=false

work very well. Resources created by the operator are not deleted/synchronized during the synchronization process.

However, the resources still appear in the interface and make the application very confusing.

[jkupferer]

We ran into this issue here while using ArgoCD to manage OpenShift authentication configuration. We create a configmap holding a CA Certificate and a secret holding some credentials, those get copied by OpenShift operators and ArgoCD finds the copies and thinks it manages them. The IgnoreExtraneous helps, but I really wish I could make the copies just not appear at all somehow.