When updating Cozystack config, dependent applications are not updated

[NickVolynkin]

• https://github.com/cozystack/cozystack/pull/818
• https://github.com/cozystack/cozystack/pull/849
• make system applications to load hash tenant config

[lllamnyp]

A lot of our apps/packages/HelmReleases rely heavily on the lookup function of the Helm templating engine. When Flux checks the status of the HelmRelease, it never renders the manifests from scratch. If something has changed in the cozy-system/cozystack configmap, these changes will not be picked up unless an administrator manually runs flux reconcile --force. Drift detection also does not help as it is concerned with watching resources created by the HelmRelease and checking them against the sh.helm... secret contain the release data. The manifests themselves are again never re-rendered.

One option would be to simply --force reconcile every HelmRelease on a timer, e.g. once every 1-5 minutes, but besides putting unnecessary load on the API server, this might also break things: there are some scenarios, where the root tenant needs to be updated first and only then the nested tenants (and then their nested tenants, etc).

To this end every tenant will now also create a configmap in its namespace, e.g. cozy-tenant-configuration-hash and every managed app (which is a HelmRelease) will take spec.valuesFrom of this configmap. The contents of the configmap will be a sha256 of .Values. tenant-root is an exception, as it sources some configuration from the configmap cozy-system/cozystack, this will be handled with some sort of if clause in the templates. In this way the reconciliation loop will now

• update the tenants and their apps when lookuped resources change
• in a breadth-first-search order, so
• child tenants will never be updated before their parents

[kvaps]

Let's implement this logic in cozystack-controller:

**option 1:** - cozystack-controller should watch for HelmReleases from `tenant` helm chart with `tenant-` prefix - caclulate hash from ( `cozystack.io/hash` annotation + values ) - write result hash to annotation to `cozystack.io/hash` for every HelmRelease in tenant namespace - if old `cozystack.io/hash` and new `cozystack.io/hash` annotation is different update also fluxcd annotation `forceReconciledAt` **concerns:** - race condition when values updated but tenant is not

option 2:

• cozystack-controller should watch for HelmReleases from tenant helm chart with tenant- prefix
  • caclulate hash from values
  • write result hash to values for every HelmRelease in tenant namespace
• cozystack-api should be updated to not show this field to the user concerns:
• race condition when values updated but tenant is not

option 3:

• cozystack-controller should watch for HelmReleases from tenant helm chart with tenant- prefix
• after cozystack-controller detects tenant chart changed should add annotation forceReconciledAt for every HelmRelease in tenant namespace concerns:
• race condition when values updated but tenant is not

option 4:

• cozystack-controller should watch for HelmReleases from tenant helm chart with tenant- prefix
• watch status for conditions.type.Ready==True
• takes history[0].configDigest
• write into values for every HelmRelease in tenant namespace
• (optional) cozystack-api should be updated to not show this field to the user concerns:
• race condition when values updated but tenant is not

option 5:

• cozystack-controller should watch for HelmReleases from tenant helm chart with tenant- prefix
• watch status for conditions.type.Ready==True
• takes history[0].configDigest
• write result hash to annotation to cozystack.io/tenant-config-digest for every HelmRelease in tenant namespace
• if old cozystack.io/hash and new cozystack.io/tenant-config-digest annotation is different update also fluxcd annotation forceReconciledAt

status:
  conditions:
  - lastTransitionTime: "2025-04-23T09:31:35Z"
    message: Helm upgrade succeeded for release tenant-root/tenant-root.v79 with chart
      [email protected]
    observedGeneration: 66
    reason: UpgradeSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-04-23T09:31:35Z"
    message: Helm upgrade succeeded for release tenant-root/tenant-root.v79 with chart
      [email protected]
    observedGeneration: 66
    reason: UpgradeSucceeded
    status: "True"
    type: Released
  helmChart: cozy-public/tenant-root-tenant-root
  history:
  - chartName: tenant
    chartVersion: 1.10.0
    configDigest: sha256:806dc06842600f15e293ae4467930c0d2d6c13876defb45462831a84a6224040
    digest: sha256:9cd5c7b034865322c90ce4b796d52d34a78fdfcd766fbe9f069f66347bb6a2c6
    firstDeployed: "2024-12-04T10:47:59Z"
    lastDeployed: "2025-04-23T09:31:34Z"
    name: tenant-root
    namespace: tenant-root
    status: deployed
    version: 79