Sometimes metrics get returned for deleted HelmReleases

[csomm]

Describe the bug

We deploy some HelmReleases only for a short time (few minutes), and after removal of the HelmRelease, the corresponding resources get deleted as expected, but sometimes metrics (with status="Unknown") get still returned for the deleted HelmRelease. This triggers alerts in our monitoring system.

As a workaround the flux2 pods can be restarted, and afterwards the metrics for the deleted HelmReleases are no longer returned.

Steps to reproduce

Deploy a HelmRelease, wait for it to become ready. Then delete the HelmRelease, and after deletion check the returned metrics. The problem seems to show up sporadically, we don't know what exactly what triggers the problem.

Expected behavior

No metrics should be returned for deleted/non-existent HelmReleases.

OS / Distro

Azure AKS K8s Cluster with flux2

Flux version

flux version 0.31.1

Flux check

► checking prerequisites ✔ Kubernetes 1.22.6 >=1.20.6-0 ► checking controllers ✔ all checks passed

We use a non-standard manifest, here are the used images: ghcr.io/fluxcd/kustomize-controller:v0.25.0 ghcr.io/fluxcd/source-controller:v0.25.6 ghcr.io/fluxcd/helm-controller:v0.21.0

Git provider

Azure

Code of Conduct

I agree to follow this project's Code of Conduct