kubernetes-logging-helm
kubernetes-logging-helm copied to clipboard
Fluentbit loading unwanted index patterns
@nickytd your changes to https://github.com/nickytd/kubernetes-logging-helm/issues/27, included some changes from my PR, but did not include these changes:
_output-opensearch-containers.conf: |-
{{ if has "containers" .Values.opensearch_dashboards.indexPatterns }}
And the same for systemd. Did you do it a different way? Without the change(s), all the data is still indexed and does not fix my original issue.
- The Kibana fix which you did include was just a bonus
You made a few comments in the issue, but I am left with the same fact. OpenSearch is eating all my space with data I do not want.
I used all the customization pieces you did mention:
Here is a pattern that demonstrates how to redirect the logs from a specific workload to its own index in fluent-bit configuration
Create a [filter](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/filter-nginx.conf) catching the workload logs and set a new tag. In this example we use kubernetes labels to identify nginx ingress controller logs and tag them with nginx
(Optional) Create a [parser](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/parser-nginx.conf) to transform the logs from lines to structured logs
Create a fluent-bit [output](https://github.com/nickytd/kubernetes-logging-helm/blob/0aa4bfd57accbf28b673d119c1e9d2c63e444e4c/chart/fluent-bit-configs/output-nginx.conf) containing the dedicated index prefix name.
If you didn't like way I had turned it off, we just need another mechanism. Do you have a proposal that I could look into to implement?