issue - telegraf-operator - MountVolume.SetUp failed for volume "telegraf-config" : secret "telegraf-XXXX" not found

[anthosz]

Hello,

Since few month, we experiment this kind of issue (50% of time when we plan an upgrade (when the pod respawn) and 20% of time during a pod reschedule (when it switch from a node to another one).

It is included in a Varnish statefulset.

Template

apiVersion: v1
kind: Secret
metadata:
  name: varnish
[...]
data:
  key: "XXXXXXX"
---
[...]
---
apiVersion: apps/v1
kind: StatefulSet
[...]
spec:
  template:
    metadata:
      annotations:
        telegraf.influxdata.com/env-secretkeyref-SECRET_VALUE: varnish.secret
        telegraf.influxdata.com/volume-mounts: '{"datas":"/datas"}'
        telegraf.influxdata.com/inputs: |+
[...]

How to reproduce

Deploy a new version or move the pod to another node.

Current behavior (randomly):

Warning FailedMount 2m23s (x242 over 7h58m) kubelet MountVolume.SetUp failed for volume "telegraf-config" : secret "telegraf-config-varnish-0" not found

Due to that, the pod cannot start.

Workaround:

Kill the workaround and the secret is well recreated.

Expected behavior:

The secret is found

Other informations

The age of secret source is more 100 days so cannot be related to this one.

But the telegraf secret seems to be recreated every time than the pod is spawn and it seems there is an issue here: the secret cannot be created so telegraf cannot spawn (unable to mount not found secret) so the pod is freezed until we terminate it.

Versions

• K8S: 1.28.4
• Telegraf: 1.28.5
• Telegraf operator: chart: 1.3.12 / APP version: 1.3.11