Logging stack as gardener-extension

[Kristian-ZH]

How to categorize this issue?

/area logging /kind enhancement /priority normal

What would you like to be added: Transform the logging stack into gardener extension and move it into dedicated repo.

Why is this needed:

• Decouple logging from the gardener itself
• Version control
• Easier maintenance

[rfranzke]

I was thinking about the same (also for the monitoring stack), but I always ended up with the problem that the components we deploy (kube-apiserver, KCM, etc.) provide the configuration for the fluent-bit / prometheus / grafana / alertmanager. It "felt" wrong to maintain these components separately while Gardener depends on them and their specific compatibility. For example, if we need to change something in the scrape configs with a newer Prometheus version then we have a version-compatibility conflict/problem between Gardener and the externalized monitoring stack. The same applies for the logging stack.

Do you have ideas/suggestions how to overcome them?

[Kristian-ZH]

Hmm, this was only an idea, we still did not invest time into this approach but now when you mentioned about this... Nope, I have no idea. Also there are configurations which are coming from the other extensions...

[rfranzke]

Also there are configurations which are coming from the other extensions...

Hm, well, that's a good point actually and it reveals that we already have the compatibility problem today. So, maybe it's not even as much of a problem because those fluent-bit/scrape config/... APIs don't change that often (actually, I've not seen that they changed at all in the past years). Maybe it's fair to still explore this approach (externalizing both monitoring and logging into an extension) a little more.

/cc @wyb1 @istvanballok (FYI, maybe you have an opinion as well because I think I remember that you'd prefer the externalized approach as well?)

[Kristian-ZH]

One of the things we should also consider is what will happen with Grafana, because now the monitoring and the logging stack use the same Grafana

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed You can:
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close

/lifecycle rotten

[gardener-robot]

@gardener-ci-robot Command /add is not available to you but only to a Maintainer, Member, Author.

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed You can:
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close

/lifecycle rotten

[rfranzke]

There was GEP-19 which discussed how to evolve the monitoring stack, and we couldn't find a sustainable approach to make this an extension. I doubt that it would be different for the logging stack. Hence, I suggest to close this issue and rather improve the data flow in the code (ref #2754).

[rfranzke]

/close in favor of https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md and https://github.com/gardener/gardener/issues/7120

[gardener-prow[bot]]

@rfranzke: Closing this issue.

In response to this:

/close in favor of https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md and https://github.com/gardener/gardener/issues/7120

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.