logging-operator operator constanly tries to reconcile fluentbitagent because it tries to update daemonset labels

Logging operator constantly reports following error:

DaemonSet.apps "rancher-logging-eks-fluentbit" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"rancher-logging-eks", "app.kubernetes.io/managed-by":"rancher-logging-eks", "app.kubernetes.io/name":"fluentbit"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

When I enable workloads recreation as mentioned in logs by setting logging objects spec.enableRecreateWorkloadOnImmutableFieldChange: true then operator constantly recreates fluentbit agent daemonsets either based on some schedule or when operator is restarted etc. Deletion of CRD and recreating anew does not solve issue.

Error seems to stem from attempt to change daemonset selector but I am not familiar enough with operator code to say why it tries to change selector.

Describe the bug: Operator constantly tries to recreate fluentbit daemonsets because it fails reconcilation Expected behaviour: No attempts at recreation of daemonsets unless FluentbitAgent CRD has changed.

Steps to reproduce the bug:

Install Rancher control plane (v2.9.0) on EKS
Add managed EKS cluster
Install Rancher Logging via Rancher apps.

Additional context: Add any other context about the problem here.

Environment details:

Kubernetes version (e.g. v1.15.2): 1.30
Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): EKS
logging-operator version (e.g. 2.1.1): 4.8.0
Install method (e.g. helm or static manifests): Rancher logging helm chart - https://github.com/rancher/charts/tree/release-v2.9/charts/rancher-logging/104.1.2%2Bup4.8.0
Logs from the misbehaving component (and any other relevant logs):

{"level":"error","ts":"2024-10-31T13:03:37Z","msg":"Reconciler error","controller":"logging","controllerGroup":"logging.banzaicloud.io","controllerKind":"Logging","Logging":{"name":"rancher-logging-eks"},"namespace":"","name":"rancher-logging-eks","reconcileID":"158a57af-91ab-4485-a2dc-055889fc2e0e","error":"failed to reconcile resource: Object has to be recreated, but refusing to remove without explicitly being told so. Use logging.spec.enableRecreateWorkloadOnImmutableFieldChange to move on but make sure to understand the consequences. As of fluentd, to avoid data loss, make sure to use a persistent volume for buffers, which is the default, unless explicitly disabled or configured differently. As of fluent-bit, to avoid duplicated logs, make sure to configure a hostPath volume for the positions through logging.spec.fluentbit.spec.positiondb. : DaemonSet.apps "rancher-logging-eks-fluentbit" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"rancher-logging-eks", "app.kubernetes.io/managed-by":"rancher-logging-eks", "app.kubernetes.io/name":"fluentbit"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable","errorVerbose":"DaemonSet.apps "rancher-logging-eks-fluentbit" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"rancher-logging-eks", "app.kubernetes.io/managed-by":"rancher-logging-eks", "app.kubernetes.io/name":"fluentbit"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\nObject has to be recreated, but refusing to remove without explicitly being told so. Use logging.spec.enableRecreateWorkloadOnImmutableFieldChange to move on but make sure to understand the consequences. As of fluentd, to avoid data loss, make sure to use a persistent volume for buffers, which is the default, unless explicitly disabled or configured differently. As of fluent-bit, to avoid duplicated logs, make sure to configure a hostPath volume for the positions through logging.spec.fluentbit.spec.positiondb. \ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/[email protected]/pkg/reconciler/resource.go:515\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentbit.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentbit/fluentbit.go:149\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to reconcile resource\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentbit.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentbit/fluentbit.go:151\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  annotations:
    meta.helm.sh/release-name: rancher-logging
    meta.helm.sh/release-namespace: cattle-logging-system
  labels:
    app.kubernetes.io/instance: rancher-logging
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rancher-logging
    app.kubernetes.io/version: 4.8.0
    helm.sh/chart: rancher-logging-104.1.1_up4.8.0
  name: rancher-logging-eks

spec:
  disableKubernetesFilter: true
  extraVolumeMounts:
    - destination: /var/log/messages
      readOnly: true
      source: /var/log/messages
  image:
    repository: rancher/mirrored-fluent-fluent-bit
    tag: 2.2.0
  inputTail:
    Buffer_Chunk_Size: 1MB
    Buffer_Max_Size: 5MB
    Parser: syslog
    Path: /var/log/messages
    Tag: eks
  nodeSelector:
    kubernetes.io/os: linux
  podPriorityClassName: system-cluster-critical
  tolerations:
    - operator: Exists

/kind bug

Oct 31 '24 13:10 siimaus

@siimaus thanks for the report and sorry for the long delay! Have you tried to set loggins.spec. enableRecreateWorkloadOnImmutableFieldChange = true? This flag is required for the operator to let it recreate the agent daemonset in case there is a change.

Nov 25 '24 10:11 pepov

@siimaus sorry I was mixing things up. Can you please provide your logging resource as well? If you let the resource to be recreated what is the difference between the original and the recreated resource exactly?

Dec 02 '24 12:12 pepov

@pepov I'm also seeing this happening when I have multiple loggings, and consequently multiple fluentbitagents.

If I enable spec.enableRecreateWorkloadOnImmutableFieldChange it recreates the Daemonset with a different spec.selector, specifically setting selector.Matchlabels["app.kubernetes.io/managed-by:"] to the other FluentBitAgent, not the one which is actually responsible for this Daemonset.

Dec 17 '24 17:12 jbiers

Do you use loggingRef to isolate your fluentbitagents? Can you describe your setup a bit more in general?

Dec 17 '24 17:12 pepov

Bug here is that there is multiple implementations how loggingRef is gotten for comparisons in go code, or in k8s via selector, or in actual logic like hostnames.

In some cases value is from loggingRef value, name, empty or default. You just need to set loggingRef for all components you use and support it, for example FluentbitAgent, Logging, ClusterOutput, ClusterFlow, to fix this issue.

These use logginRef in both directions, https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/controllers/logging/logging_controller.go#L640-L643 and https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/controllers/logging/logging_controller.go#L546-L565

But secrets have their own implementation, https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/controllers/logging/logging_controller.go#L572-L585

ClusterFlow and ClusterOutput use loggingRef, https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/model/repository.go#L151-L159 https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/model/repository.go#L171-L179

As does FluentbitAgent, https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/model/repository.go#L326-L334

LoggingRoutes use source and loggingRef https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/model/repository.go#L428-L436

Fluentd uses name instead of loggingRef https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/sdk/logging/api/v1beta1/logging_types.go#L556-L563 https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/sdk/logging/api/v1beta1/logging_types.go#L604-L605

ServiceMonitor uses name, https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/fluentbit/service.go#L60-L76

Secrets use loggingRef or "default" https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/fluentd/outputsecret.go#L35-L39

Main uses loggingRef if set https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/main.go#L333-L334

Fluentbit uses name https://github.com/kube-logging/logging-operator/blob/c6bf514d68b5f5711d451e6b76f7f6c46ee7cfbb/pkg/resources/fluentbit/fluentbit.go#L54-L65

Jan 23 '25 10:01 maage

Yes, unfortunately this has been handled differently throughout the code and haven't been fixed since. I think it will be non-trivial to fix in a compatible manner. For now can someone give as a reproducible example?

Feb 03 '25 08:02 pepov

@pepov having loggingRef in values.yaml works for me...

logging:

  # -- Logging resources are disabled by default
  enabled: false

  # -- Reference to the logging system. Each of the loggingRefs can manage a fluentbit daemonset and a fluentd statefulset.
  loggingRef: "xxxxxxx"

Feb 10 '25 05:02 skanakal

I had the same problem. Changin spec.enableRecreateWorkloadOnImmutableFieldChange: true solved the issue

Feb 17 '25 14:02 pszczypta-autopay

Same issue here. We use Rancher as well. enableRecreateWorkloadOnImmuatableFieldChange=true didn't help. Every time a flow was deployed, the fluentbit pods were restarted, leading often (but not always) to a missing OUTPUT block in the fluentbit secret data. Rancher helm chart version 104.1.1+up4.8.0, which uses logging operator 4.8.0, is affected. Version 103.1.1+up4.4.0 (logging operator 4.4.0) works fine.

May 22 '25 05:05 vickysgd

can someone provide me a close to complete configuration on discord or slack (can send me privately if that helps) https://kube-logging.dev/docs/community/

May 22 '25 08:05 pepov

This is what we have. We have yet to migrate to a separate FluentdConfig object so that portion is still in the Logging object. I would not have expected that to be the issue though when comparing to others.

FluentBitAgent

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  annotations:
    meta.helm.sh/release-name: rancher-logging
    meta.helm.sh/release-namespace: cattle-logging-system
    objectset.rio.cattle.io/id: default-rancher-logging-rke2
  labels:
    app.kubernetes.io/instance: rancher-logging
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rancher-logging
    app.kubernetes.io/version: 4.8.0
    helm.sh/chart: rancher-logging-104.1.0_up4.8.0
    objectset.rio.cattle.io/hash: 4dceeb36e9800adfd717f17494d3e67836d595d9
  name: rancher-logging-kube-audit
spec:
  disableKubernetesFilter: true
  dnsConfig:
    options:
    - name: ndots
      value: "3"
  extraVolumeMounts:
  - destination: /kube-audit-logs
    readOnly: true
    source: /var/lib/rancher/rke2/server/logs
  filterKubernetes:
    Buffer_Size: "0"
  filterModify:
  - rules:
    - Add:
        key: type
        value: kube-audit
  image:
    repository: rancher/mirrored-fluent-fluent-bit
    tag: 2.2.0
  inputTail:
    Mem_Buf_Limit: 75MB
    Parser: json
    Path: /kube-audit-logs/audit.log
    Tag: kube-audit
    multiline.parser:
    - cri
  loggingRef: containerservices
  metrics:
    serviceMonitor: true
    serviceMonitorConfig:
      additionalLabels:
        owner: containerservices
  nodeSelector:
    node-role.kubernetes.io/master: "true"
  podPriorityClassName: system-cluster-critical
  resources:
    limits:
      memory: 512Mi
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
  - effect: NoSchedule
    key: node-role.kubernetes.io/controlplane
    value: "true"
  - effect: NoExecute
    key: node-role.kubernetes.io/etcd
    operator: Exists

Logging

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  annotations:
    meta.helm.sh/release-name: rancher-logging
    meta.helm.sh/release-namespace: cattle-logging-system
    objectset.rio.cattle.io/id: default-rancher-logging-rke2
  labels:
    app.kubernetes.io/instance: rancher-logging
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rancher-logging
    app.kubernetes.io/version: 4.8.0
    helm.sh/chart: rancher-logging-104.1.0_up4.8.0
    objectset.rio.cattle.io/hash: 4dceeb36e9800adfd717f17494d3e67836d595d9
  name: rancher-logging-kube-audit
spec:
  controlNamespace: cattle-logging-system
  enableRecreateWorkloadOnImmutableFieldChange: true
  fluentd:
    bufferStorageVolume:
      pvc:
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
    configReloaderImage:
      repository: rancher/mirrored-jimmidyson-configmap-reload
      tag: v0.4.0
    disablePvc: false
    image:
      repository: rancher/mirrored-banzaicloud-fluentd
      tag: v1.14.6-alpine-5
    livenessProbe:
      initialDelaySeconds: 300
      periodSeconds: 30
      tcpSocket:
        port: 24240
    logLevel: info
    metrics:
      serviceMonitor: true
      serviceMonitorConfig:
        additionalLabels:
          owner: containerservices
    podPriorityClassName: system-cluster-critical
    readinessDefaultCheck:
      bufferFileNumber: true
      bufferFileNumberMax: 10000
      bufferFreeSpace: true
      bufferFreeSpaceThreshold: 90
      failureThreshold: 1
      initialDelaySeconds: 5
      periodSeconds: 30
      successThreshold: 3
      timeoutSeconds: 10
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 10m
        memory: 1Gi
    scaling:
      drain:
        enabled: true
      replicas: 2
    security:
      podSecurityContext:
        fsGroup: 101
        runAsNonRoot: true
        runAsUser: 100
  loggingRef: containerservices
  watchNamespaces:
  - cattle-logging-system

Here is the error

"DaemonSet.apps \"rancher-logging-kube-audit-fluentbit\" is invalid: 
  spec.selector: 
    Invalid value: 
      v1.LabelSelector{
        MatchLabels:map[string]string{
          \"app.kubernetes.io/instance\":\"rancher-logging-kube-audit\",
          \"app.kubernetes.io/managed-by\":\"rancher-logging-kube-audit\",
          \"app.kubernetes.io/name\":\"fluentbit\"
        },
        MatchExpressions:[]v1.LabelSelectorRequirement(nil)
      }: field is immutable"

{
  "level": "error",
  "ts": "2025-07-02T14:38:33Z",
  "logger": "logging.fluentbit.reconciler",
  "msg": "failed to update resource, trying to recreate",
  "logging": "rancher-logging-kube-audit",
  "fluentbitagent": "rancher-logging-kube-audit",
  "name": "rancher-logging-kube-audit-fluentbit",
  "namespace": "cattle-logging-system",
  "apiVersion": "apps/v1",
  "kind": "DaemonSet",
  "name": "rancher-logging-kube-audit-fluentbit",
  "namespace": "cattle-logging-system",
  "apiVersion": "apps/v1",
  "kind": "DaemonSet",
  "error": "DaemonSet.apps \"rancher-logging-kube-audit-fluentbit\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/instance\":\"rancher-logging-kube-audit\", \"app.kubernetes.io/managed-by\":\"rancher-logging-kube-audit\", \"app.kubernetes.io/name\":\"fluentbit\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable",
  "stacktrace": "github.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/[email protected]/pkg/reconciler/resource.go:481\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentbit.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentbit/fluentbit.go:149\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"
}

Jul 02 '25 16:07 atsai1220

We've tried to reproduce the issue locally but without any luck...

Do you have any other logging resources or FluentBitAgents in the system? Is there any specific reason why you use LoggingRef's?

Jul 09 '25 10:07 csatib02

We've tried to reproduce the issue locally but without any luck...

Do you have any other logging resources or FluentBitAgents in the system? Is there any specific reason why you use LoggingRef's?

We're using loggingRefs because this is a multi-tenant environment and we have other users deploying a separate group of Logging objects.

Jul 10 '25 02:07 atsai1220

i am also gettings similar error. I added enableRecreateWorkloadOnImmutableFieldChange = true in helm values but it is not reflecting in crd.

logging-operator version: 4.10.0

{"level":"error","ts":"2025-07-10T06:33:38Z","msg":"Reconciler error","controller":"logging","controllerGroup":"logging.banzaicloud.io","controllerKind":"Logging","Logging":{"name":"rancher-logging-root"},"namespace":"","name":"rancher-logging-root","reconcileID":"0cd10d29-f990-4c57-b34e-b73318cd56e7","error":"failed to reconcile resource: Object has to be recreated, but refusing to remove without explicitly being told so. Use logging.spec.enableRecreateWorkloadOnImmutableFieldChange to move on but make sure to understand the consequences. As of fluentd, to avoid data loss, make sure to use a persistent volume for buffers, which is the default, unless explicitly disabled or configured differently. As of fluent-bit, to avoid duplicated logs, make sure to configure a hostPath volume for the positions through logging.spec.fluentbit.spec.positiondb. : DaemonSet.apps "rancher-logging-fluentbit" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"rancher-logging", "app.kubernetes.io/managed-by":"rancher-logging-root", "app.kubernetes.io/name":"fluentbit"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable","errorVerbose":"DaemonSet.apps "rancher-logging-fluentbit" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"rancher-logging", "app.kubernetes.io/managed-by":"rancher-logging-root", "app.kubernetes.io/name":"fluentbit"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\nObject has to be recreated, but refusing to remove without explicitly being told so. Use logging.spec.enableRecreateWorkloadOnImmutableFieldChange to move on but make sure to understand the consequences. As of fluentd, to avoid data loss, make sure to use a persistent volume for buffers, which is the default, unless explicitly disabled or configured differently. As of fluent-bit, to avoid duplicated logs, make sure to configure a hostPath volume for the positions through logging.spec.fluentbit.spec.positiondb. \ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/[email protected]/pkg/reconciler/resource.go:515\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentbit.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentbit/fluentbit.go:149\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:304\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nfailed to reconcile resource\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentbit.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentbit/fluentbit.go:151\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:304\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}

Jul 10 '25 06:07 jothiprasath-cdmx

We've tried to reproduce the issue locally but without any luck... Do you have any other logging resources or FluentBitAgents in the system? Is there any specific reason why you use LoggingRef's?

We're using loggingRefs because this is a multi-tenant environment and we have other users deploying a separate group of Logging objects.

Do the other logging resources look the similar? Do they have a different logging ref other than containerservices?

Jul 10 '25 07:07 pepov