Sidecar Container not starting

[shaunfink]

HI all

We're using version 2.7.0 of the official sumologic helm chart for deploying sumo to our EKS clusters. We've found a situation where we've had to make use of the tailing sidecar to ingest logs from a container for a service, as these logs are not written to stdout/stderr.

As the sumo chart has this bundled in as a dependant chart, it seems like the sidecar image is not working as expected, but is also running a MUCH older version of the chart (version 0.3.2, based on what I can see in Chart.yaml)

I get the following in logs when running the sidecar

[2022/07/01 09:42:11] [error] [sqldb] cannot open database /tailing-sidecar/var/fluent-bit-state.db
[2022/07/01 09:42:11] [error] [input:tail:tail.0] could not open/create database
[2022/07/01 09:42:11] [error] Failed initialize input tail.0
[2022/07/01 09:42:11] [error] [lib] backend failed

Overriding the version of both the sidecar to version 0.5.0 seems to work a treat though.

Configs I'm defining in values.yaml for the sumo chart are the following:

tailing-sidecar-operator:
  enabled: true
  
  sidecar:
    image:
      tag: 0.5.0

If I do overrides for later versions of the operator (pretty much anything up from 0.3.2), I see loads of these issues in operator logs (which look to relate to #168:

E0701 10:10:45.493493       1 leaderelection.go:325] error retrieving resource lock monitoring/7b555970.sumologic.com: leases.coordination.k8s.io "7b555970.sumologic.com" is forbidden: User "system:serviceaccount:monitoring:tailing-sidecar-service-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "monitoring"

Reverting all these configs to use the 0.3.2 operator, and the 0.5.0 sidecar looks to work.

[kasia-kujawa]

In sumologic-kubernetes-collection the version of tailing sidecar operator is quite old because the newest version contains breaking changes (https://github.com/SumoLogic/sumologic-kubernetes-collection/issues/1739) I don't recommend of using older operator with newer version of sidecar image but testing the configuration that you mention I haven't notice any issue.

Could you share the configuration that you prepared for tailing sidecar operator and specification of the Pod with changes made by tailing sidecar operator?

If you want to use newer version of the Tailing Sidecar Operator you can install it as separated helm chart without sumologic-kubernetes-collection, details: https://github.com/SumoLogic/tailing-sidecar#tldr

#168 may be observed is some clusters, now tailing-sidecar-leader-election-role role must be manually extended, following permissions must be add:

- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - list
  - update

[sumo-drosiek]

This would be fixed by #377

[kasia-kujawa]

I'm closing this issue, @shaunfink if you still observe the issue, please reopen it ;)