Agent Pods not reloading renamed or new PolicyID

[barkbay]

While working on https://github.com/elastic/cloud-on-k8s/issues/7290 I noticed that renaming a policy ID does not trigger a "restart" of the Agent Pods. The FLEET_ENROLLMENT_TOKEN is provided as an environment variable, and therefore requires a new Pod to be created to read the new value.

apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: "2024-10-18T12:42:54Z"
  labels:
    agent.k8s.elastic.co/name: elastic-agent
    common.k8s.elastic.co/type: agent
    eck.k8s.elastic.co/credentials: "true"
  name: elastic-agent-agent-envvars
  namespace: elastic
stringData:
  FLEET_ENROLLMENT_TOKEN: REDACTED // This is going to be updated correctly by the Agent controller
type: Opaque

apiVersion: v1
kind: Pod
metadata:
  annotations:
    agent.k8s.elastic.co/config-hash: "803455129"
    openshift.io/scc: privileged
  creationTimestamp: "2024-10-18T12:42:54Z"
  generateName: elastic-agent-agent-
  labels:
    agent.k8s.elastic.co/name: elastic-agent
    agent.k8s.elastic.co/version: 8.15.0
    common.k8s.elastic.co/type: agent
    controller-revision-hash: 6ccd744885
    pod-template-generation: "1"
  name: elastic-agent-agent-hjn5b
  namespace: elastic
spec:
  containers:
    - name: FLEET_ENROLLMENT_TOKEN
      valueFrom:
        secretKeyRef:
          key: FLEET_ENROLLMENT_TOKEN // Not reloaded without a restart
          name: elastic-agent-agent-envvars
          optional: false

Note that one other problem is that the "old" policies are never deleted and the related tokens remain valid.