fluent-bit

fluent-bit copied to clipboard

This provides the ability to have the filter_kubernetes plugin fetch namespace labels & annotations and add them to the record under a kubernetes_namespace key.

• Added 3 new configuration options:

  • kube_meta_namespace_cache_ttl - FLB_CONFIG_MAP_TIME - the TTL for the namespace cache
  • namespace_annotations - FLB_CONFIG_MAP_BOOL - On if you want namespace annotations added to the record. Default: Off
  • namespace_labels - FLB_CONFIG_MAP_BOOL - On if you want namespace labels added to the record. Default: Off

• Fetching namespace metadata can not use kubelet so the shared variables around kubelet vs kubernetes api were completely split in this patch. This allows us to still use kubelet for pod information if Use_Kubelet On is set, while still being able to fetch namespace metadata from the upstream kubernetes api.

  • if neither namespace_labels or namespace_annotations are On then fetching of namespace metadata from the kubrenetes api will NOT occur.

• a second cache (flb_hash_table) and configuration for it's ttl was added specifically for caching namespace information. I found this important as you may want to refresh namespace label information every few minutes (in case namespace labels/annotations change), while only fetching new pod metadata if the cache is full. I defaulted the namespace cache to a 15min ttl.

• metadata fetching / extraction for pod vs namespace metadata is fairly separate intentionally as based on how this Pull Request goes, i would be willing to add the ability to add namespace metadata to non-pod logs (ie the ability to enrich kubernetes_events input with the same kubernetes_namespace data)

Closes #6544 Addresses #3865

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [X] Example configuration file for the change

[FILTER]
    Name kubernetes
    Match kube*
    K8S-Logging.Parser Off
    K8S-Logging.Exclude Off
    Use_Kubelet On
    Buffer_Size  1MB
    Annotations Off
    namespace_labels  On
    namespace_annotations On

• [ ] Debug log output from testing the change

• [X] Attached Valgrind output that shows no leaks or memory corruption was found

==18850== Memcheck, a memory error detector
==18850== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==18850== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==18850== Command: ./bin/flb-rt-filter_kubernetes
==18850== 
Test kube_core_base...                          [ OK ]
Test kube_core_no_meta...                       [ OK ]
Test kube_core_unescaping_text...               [ OK ]
Test kube_core_unescaping_json...               [ OK ]
Test kube_options_use-kubelet_enabled_json...   [ OK ]
Test kube_options_use-kubelet_disabled_json...  [ OK ]
Test kube_options_merge_log_enabled_text...     [ OK ]
Test kube_options_merge_log_enabled_json...     [ OK ]
Test kube_options_merge_log_enabled_invalid_json... [ OK ]
Test kube_options_merge_log_disabled_json...    [ OK ]
Test kube_options_merge_log_trim_enabled_json... [ OK ]
Test kube_options_merge_log_trim_disabled_json... [ OK ]
Test kube_options_merge_log_key_json...         [ OK ]
Test kube_options_keep_log_enabled_json...      [ OK ]
Test kube_options_keep_log_disabled_json...     [ OK ]
Test kube_options_k8s_logging_parser_disabled_text_stdout... [ OK ]
Test kube_options_k8s_logging_parser_disabled_text_stderr... [ OK ]
Test kube_options_k8s_logging_exclude_disabled_text_stdout... [ OK ]
Test kube_options_k8s_logging_exclude_disabled_text_stderr... [ OK ]
Test kube_annotations_invalid_text...           [ OK ]
Test kube_annotations_parser_regex_with_time_text... [ OK ]
Test kube_annotations_parser_regex_with_time_invalid_text_1... [ OK ]
Test kube_annotations_parser_json_with_time_json... [ OK ]
Test kube_annotations_parser_json_with_time_invalid_json_1... [ OK ]
Test kube_annotations_parser_invalid_text_stdout... [ OK ]
Test kube_annotations_parser_invalid_text_stderr... [ OK ]
Test kube_annotations_parser_stdout_text_stdout... [ OK ]
Test kube_annotations_parser_stdout_text_stderr... [ OK ]
Test kube_annotations_parser_stderr_text_stdout... [ OK ]
Test kube_annotations_parser_stderr_text_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_1_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_1_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_2_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_2_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_3_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_3_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_4_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_4_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_5_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_5_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_1_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_1_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_2_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_2_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_3_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_3_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_4_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_4_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_5_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_5_stderr... [ OK ]
Test kube_annotations_exclude_default_text...   [ OK ]
Test kube_annotations_exclude_invalid_text_stdout... [ OK ]
Test kube_annotations_exclude_invalid_text_stderr... [ OK ]
Test kube_annotations_exclude_stdout_text_stdout... [ OK ]
Test kube_annotations_exclude_stdout_text_stderr... [ OK ]
Test kube_annotations_exclude_stderr_text_stdout... [ OK ]
Test kube_annotations_exclude_stderr_text_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_4_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_4_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_4_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_4_stderr... [ OK ]
Test kube_systemd_logs...                       [ OK ]
SUCCESS: All unit tests have passed.
==18850== 
==18850== HEAP SUMMARY:
==18850==     in use at exit: 0 bytes in 0 blocks
==18850==   total heap usage: 277,552 allocs, 277,552 frees, 102,355,837 bytes allocated
==18850== 
==18850== All heap blocks were freed -- no leaks are possible
==18850== 
==18850== For lists of detected and suppressed errors, rerun with: -s
==18850== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [ ] Run local packaging test showing all targets (including any new ones) build.
• [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [X] Documentation required for this feature

https://github.com/fluent/fluent-bit-docs/pull/1280

Backporting

• [ ] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Added a second commit for the following... thought it was probably a good idea to include at least one test 😄 :

filter_kubernetes: Add Tests for namespace labels & annotations
    - also updates error text for kubelet vs kube api upstream errors
    - get_namespace_api_server_info passes in meta->namespace
      instead of ctx->namespace which may not be set at time of call

Initial test run failed due to missing test file. I didn't catch the .gitignore ignores *.log :). Just pushed an amended 2nd commit with the missing file. 🤞

@ryanohnemus I wonder if we can add some integration tests around the K8S filter to help prove this as well in the "real world" :)

They run on GKE currently anyway: https://github.com/fluent/fluent-bit-ci/tree/main/tests

@patrick-stephens I created https://github.com/fluent/fluent-bit-ci/pull/121, but this is checking only the currently available pod labels. I assume this PR needs to be merged in first before I can add the namespace labels check.

@patrick-stephens I created fluent/fluent-bit-ci#121, but this is checking only the currently available pod labels. I assume this PR needs to be merged in first before I can add the namespace labels check.

We probably will have to unless we add a conditional test that is skipped when the filter help does not show the new options. In which case we can merge the integration test first then re-run it for this PR to confirm all good, then a separate int test PR update to be added after this PR is merged. We can run int tests with the specific PR image from this PR as well to confirm it would pass.

@patrick-stephens - just amended the last commit comment and force pushed for it to restart the tests here with the new ci

@ryanohnemus thanks for your sterling effort on this - both on this PR and all the supporting changes for testing/docs. The changes here need a bit of time to review.

@patrick-stephens I really appreciate the feedback and the help on getting the new k8s ci tests added. They are finally passing! 🙌

I will work on the extended CI checks for k8s that contains the namespace labels checks as well as some additional tests I think should be added along with them.

We can run int tests with the specific PR image from this PR as well to confirm it would pass.

Do you have an example of how to run with the PR image? I assume I change theFLUENTBIT_IMAGE_REPOSITORY & FLUENTBIT_IMAGE_TAG envs, but not exactly sure where I get those details from the PR build's image?

Do you have an example of how to run with the PR image? I assume I change theFLUENTBIT_IMAGE_REPOSITORY & FLUENTBIT_IMAGE_TAG envs, but not exactly sure where I get those details from the PR build's image?

The integration tests already pass the details over: https://github.com/fluent/fluent-bit/blob/18e5eda4b644723fcfbe6a46524de8430f856fe5/.github/workflows/pr-integration-test.yaml#L48 The image is first built then passed to the workflow - it's a shame we have duplicate workflows.

@patrick-stephens thanks again for the pointer. I've added extended tests here: https://github.com/fluent/fluent-bit-ci/pull/127 and tested against this PR's image.

Let me know if there's more that is needed!

Apologies for all the empty force pushed commits, I was trying (and failing) to get a kubelet enabled CI test running. Thank you again @patrick-stephens for your help (and patience) in that!

@edsiper as far as progressing this PR forward, is there any estimate/milestone this can be attached to? I'm looking forward to getting this across the finish line!

This feature is eagerly awaited in kube-logging/logging-operator#1148 😄 Anything we can do to move this forward?

@ryanohnemus I think we can link out to tests proving this now? That will demonstrate both the current behaviour working and this new behaviour right? If you can link that with some example output as well to show @edsiper this is well tested that will help.

@patrick-stephens @edsiper - yes, i added CI tests for the kubernetes_filter here: https://github.com/fluent/fluent-bit-ci/blob/main/tests/kubernetes-plugins/full.bats and we have an example of those passing (also attached to this PR via the checks) here: https://github.com/fluent/fluent-bit/actions/runs/7654984585/job/20860421190?pr=8279

@ryanohnemus post the log contents as well for posterity as they'll expire eventually

@patrick-stephens understood!

---

1. namespace_labels only test, where a label this_is_a_namespace_label: true exists. The filter config is:

    [FILTER]
        Name             kubernetes
        Alias            k8s_namespace_labels_only
        Match            kube.*k8s-namespace-label-tester*
        Kube_URL         https://kubernetes.default.svc:443
        Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix  kube.var.log.containers.
        Merge_Log        Off
        Annotations      Off
        Labels           Off
        Namespace_labels On
        Namespace_annotations Off

And the output is:

{"date":1706799783.840397,"time":"2024-02-01T15:03:03.840397339Z","stream":"stdout","_p":"F","log":"hello world from k8s-namespace-label-tester","kubernetes_namespace":{"name":"test","labels":{"kubernetes.io/metadata.name":"test","this_is_a_namespace_label":"true"}}}

You should see a kubernetes_namespace (namespace labels) map and no kubernetes (pod labels) map in this one.

---

2. namespace labels and pod labels test

config:

    [FILTER]
        Name             kubernetes
        Alias            k8s_namespace_and_pod_labels_without_kubelet
        Match            kube.*k8s-pod-and-namespace-label-tester*
        Kube_URL         https://kubernetes.default.svc:443
        Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix  kube.var.log.containers.
        Merge_Log        Off
        Annotations      Off
        Labels           On
        Namespace_labels On

output:

{"date":1706799799.653302,"time":"2024-02-01T15:03:19.653302082Z","stream":"stdout","_p":"F","log":"hello world from k8s-pod-and-namespace-label-tester","kubernetes":{"pod_name":"k8s-pod-and-namespace-label-tester","namespace_name":"test","pod_id":"2203df11-9417-47f6-b4de-e698191cce14","labels":{"this_is_a_test_label":"true"},"host":"kind-worker2","container_name":"k8s-pod-and-namespace-label-tester","docker_id":"616979d997d0a7e4c517c15da07424506e9e8d3410c3c55dd2f69e590aab8c6b","container_image":"docker.io/library/alpine:latest"},"kubernetes_namespace":{"name":"test","labels":{"kubernetes.io/metadata.name":"test","this_is_a_namespace_label":"true"}}}

You should see both a kubernetes_namespace (namespace labels) map and a kubernetes (pod labels) map in this one.

ci test failure looks like a flake on gke runner, and mac os unit test is a failure on flb-it-log

Yeah @ryanohnemus I'm happy you've done your best with the flakes :) There's a fair bit of stuff to review but this is in the pipeline

thank you!

Well done @ryanohnemus!