Kubernetes filter is not able to get data for multiple tail input

[Ripurwar80]

Bug Report

fluent bit image version: 2.1

I have multiple tail inputs which are pulling data from different log files and tagging them under different tags for filtering with different regex.

for example [INPUT] Name tail Tag kube.amp-system.* Path /var/log/containers/amp-system.log Parser docker Refresh_Interval 5 Mem_Buf_Limit 10MB Skip_Long_Lines On DB /var/log/flb_tail-amp-system.db

[INPUT]
    Name             tail
    Tag              kube.out-nginx.*
    Path             /var/log/containers/*_out-nginx_*.log
    Parser           docker
    Refresh_Interval 5
    Mem_Buf_Limit    10MB
    Skip_Long_Lines  On
    DB               /var/log/flb_tail-out-nginx.db

[INPUT]
    Name             tail
    Tag              kube.sys-wd-tenant.*
    Path             /var/log/containers/*_*-design*.log
    Parser           docker
    Refresh_Interval 5
    Mem_Buf_Limit    10MB
    Skip_Long_Lines  On
    DB               /var/log/flb_tail-pto-design.db

for these input i need to enrich them with Kubernetes data so i have created a Kubernetes filter as below.

[FILTER] Name kubernetes Match kube.* Kube_Tag_Prefix kube.*.var.log.containers. Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Merge_Log On K8S-Logging.Parser On Buffer_Size 0 Cache_Use_Docker_Id On Labels On Merge_Log_Key log_processed

what I read in the documentation is : If the configuration property Kube_Tag_Prefix was configured (available on Fluent Bit >= 1.1.x), it will use that value to remove the prefix that was appended to the Tag in the previous Input section. Note that the configuration property defaults to kube.var.logs.containers.

so, in that case, I have modified the input tail tag from kube.* to kube.[something].*

According to it, I'm modifying my Kube_Tag_Prefix, to accept the changes into tail input Kube_Tag_Prefix kube.*.var.log.containers.

i have also tried using regex Kube_Tag_Prefix ^kube.([^.]+).var.log.container.$

but it is taking only one tail input into consideration and all other input are not getting enrich with kubernetes metadata.

please help.

[patrick-stephens]

Did we discuss this in Slack the other day?

I don't think you can use a wildcard in the kube tag prefix. Using debug logging should show you the API server responses which likely will show it's just including the actual wildcard character and not expanding it.

[Ripurwar80]

That’s the problem, but I used ^kube.([^.]+).var.log.containers.$ as kube_tag_prefix and it was able to match with one of the tail input but discarder others. i wonder if thats the issue with fluent bit

[patrick-stephens]

I think the prefix is only supposed to be a simple string, not a full blown regex engine.

[Ripurwar80]

I got your point, but again what’s the use of having the capability of multiple tail inputs, when only one input can be enrich with kube metadata. also if it’s able to catch one of the tail input tag and then why not others, in general production will have multiple application running we can’t get all the logs under one input as it will not possible to filter specific logs. there has to be some solution for it, i'm just waiting for someone to answer

[Ripurwar80]

hey @patrick-stephens , I created a custom parser and used it in Regex_Parser, i can see in the debug mode the request sent to api server with correct ns and pod name and the response in 200. but still the l'm not seeing kuberntes values added to the logs.

any idea on this

[patrick-stephens]

What does stdout output show?

[Ripurwar80]

thanks for the reply @patrick-stephens we don’t have a stdout we are sending our logs to Splunk. [OUTPUT] Name splunk Match * Host ${SPLUNK_ENDPOINT} Port 443 Splunk_Token ${SPLUNK_TOKEN} Splunk_Send_Raw On TLS On TLS.Verify On TLS.Debug 1

[patrick-stephens]

Yes, I'd say stdout to see what exactly there is. Maybe Splunk is dropping it so verify first

[Ripurwar80]

@patrick-stephens , the fun part is everything working fine with the fluentbit 1.3.3, but not with the 2.1, i went through the release note as well not found anything

[patrick-stephens]

I mean that's quite a big jump, I would debug via stdout first to confirm.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.