invalid pattern for given tag tail + Kubernetes

[VincentSchmid]

Bug Report

Describe the bug When I set the tail input to (as described in the tail documentation):

Tag               kube.<namespace_name>.<pod_name>.<container_name>
Path              /var/log/containers/*.log
Tag_Regex        (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-

I only get log spammed:

[ warn] [filter:kubernetes:kubernetes.0] invalid pattern for given tag kube.kube-system.fluent-bit-q2kxj.fluent-bit

Probably I'm missing something. The documentation is clear, but I might need an example of how this is supposed to work.

To Reproduce

• Steps to reproduce the problem:

[SERVICE]
    Flush         1
    Log_Level     info
    Daemon        off
    Parsers_File  /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    
[INPUT]
    Name              tail
    Tag               kube.<namespace_name>.<pod_name>.<container_name>
    Path              /var/log/containers/*.log
    Tag_Regex         (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
    Parser            cri
    DB                /var/log/containers/fluentbit_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   Off
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    Merge_Log_Trim      On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Annotations         Off
    Labels              On

[FILTER]
    Name          nest
    Match         kube.*
    Operation     lift
    Nested_under  kubernetes
    Add_prefix     kubernetes_

[FILTER]
    Name          nest
    Match         kube.*
    Operation     lift
    Nested_under  kubernetes_labels
    Add_prefix     kubernetes_labels_

[OUTPUT]
    name    stdout
    match   kube.ns1.*
    Format  json

Expected behavior I would like to adjust the log tag behaviour so I can have this tag pattern: kube.<namespace_name>.<pod_name>.<container_name> I would assume that the Kubernetes logs pass through the Kubernetes filter where they get enhanced and then only the logs for the given namespace are printed to the console.

Your Environment I'm deploying fluent-bit using helm: helm install fluent-bit fluent/fluent-bit --version 0.46.2 -f helm/values_dev.yaml --namespace kube-system

• Version used: 3.0.1
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): tested on: docker-desktop kubernetes & aks

Additional context I'm trying to split the logs by namespace, to send the logs for each namespace to a different table in my log analytics workspace.

[VincentSchmid]

Figured it out! This issue helped me solve it: #1092

Not entirely sure, but maybe the documentation needs to be clarified?

[PARSER]
    Name    kube-tag
    Format  regex
    Regex   ^(?<namespace_name>[^_]+)\.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?<container_name>.+)

[INPUT]
    Name              tail
    Tag               kube.<namespace_name>.<pod_name>.<container_name>
    Path              /var/log/containers/*.log
    Tag_Regex         (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-[a-z0-9]{64}\.log$
    Parser            cri
    DB                /var/log/containers/fluentbit_kube.db
    Mem_Buf_Limit     10MB
    Skip_Long_Lines   Off
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.
    Regex_Parser        kube-tag
    Merge_Log           On
    Merge_Log_Trim      On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Annotations         Off
    Labels              On

[OUTPUT]
    name    stdout
    match   kube.ns1.*
    Format  json

[patrick-stephens]

Please submit a PR to update the docs if you can, it's a good way to easily contribute to OSS: https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/inputs/tail.md

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.