kubernetes-logging-helm icon indicating copy to clipboard operation
kubernetes-logging-helm copied to clipboard

Fluentbit loading unwanted index patterns

Open dfishburn opened this issue 2 years ago • 0 comments

@nickytd your changes to https://github.com/nickytd/kubernetes-logging-helm/issues/27, included some changes from my PR, but did not include these changes:

  _output-opensearch-containers.conf: |-
  {{ if has "containers" .Values.opensearch_dashboards.indexPatterns }}

And the same for systemd. Did you do it a different way? Without the change(s), all the data is still indexed and does not fix my original issue.

  • The Kibana fix which you did include was just a bonus

You made a few comments in the issue, but I am left with the same fact. OpenSearch is eating all my space with data I do not want.

I used all the customization pieces you did mention:

Here is a pattern that demonstrates how to redirect the logs from a specific workload to its own index in fluent-bit configuration

Create a [filter](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/filter-nginx.conf) catching the workload logs and set a new tag. In this example we use kubernetes labels to identify nginx ingress controller logs and tag them with nginx
(Optional) Create a [parser](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/parser-nginx.conf) to transform the logs from lines to structured logs
Create a fluent-bit [output](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/output-nginx.conf) containing the dedicated index prefix name.

If you didn't like way I had turned it off, we just need another mechanism. Do you have a proposal that I could look into to implement?

dfishburn avatar May 02 '23 19:05 dfishburn