Updates to Valkey resource not reflected on all subresources

[iggy]

I'm not sure how widespread the issue is, but my particular case was labels being added on the Valkey custom resource and not appearing on the statefulset (and thus not on the pod), but they were appearing on the services (notably on the selectors for the services). This lead to a situation where none of the pods matched the selectors, so the headless service would never resolve any IPs.

For anyone who runs into something similar I'm using flux/kustomize (which adds some labels to everything it manages), but I had manually applied the config with kubectl apply -k before committing the configs for flux. So I got the config to work by applying directly with kubectl then handed it off to flux which added some labels that didn't exist when I manually applied. Those extra labels never ended up on the statefulset/pods leading to the mismatch between pod labels and service selectors.

Let me know if you need any more info.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    admission.datadoghq.com/enabled: "false"
    app.kubernetes.io/component: valkey
    app.kubernetes.io/instance: n8n-valkey
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: valkey
  name: n8n-valkey
  namespace: aipms

apiVersion: v1
kind: Service
metadata:
  labels:
    admission.datadoghq.com/enabled: "false"
    app.kubernetes.io/component: valkey
    app.kubernetes.io/instance: n8n-valkey
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: valkey
    kustomize.toolkit.fluxcd.io/name: aipms
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: n8n-valkey-headless
  namespace: aipms
  ...
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: tcp-valkey
    port: 6379
    protocol: TCP
    targetPort: tcp-valkey
  - name: tcp-valkey-bus
    port: 16379
    protocol: TCP
    targetPort: tcp-valkey-bus
  publishNotReadyAddresses: true
  selector:
    admission.datadoghq.com/enabled: "false"
    app.kubernetes.io/component: valkey
    app.kubernetes.io/instance: n8n-valkey
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: valkey
#  these were added after the initial deploy and never ended up on the statefulset/pods
    kustomize.toolkit.fluxcd.io/name: aipms   <--- 
    kustomize.toolkit.fluxcd.io/namespace: flux-system   <---
  sessionAffinity: None
  type: ClusterIP

[dmolik]

Interesting thanks! The label function has testing. I'll get to the root cause of this as soon as I can

[dmolik]

this seems to be fixed with v0.0.59

[ClemaX]

There seems to be an issue in upsertStatefulSet where changes to labels are silently ignored due to this check not taking into account changed labels on the owning Valkey CR.

This causes the service selectors to become invalid if labels are added or modified on the owning Valkey CR after initial creation, as these are upserted without any checks for differences.