FluentBit regex is matched, but I can't find my field, such as level etc.

[zhangzx1996]

Bug Report

Describe the bug

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}

• Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

• Version used: 3.0.3
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): K8S
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[zhangzx1996]

This is my yaml file: kind: ConfigMap apiVersion: v1 metadata: name: fluent-bit-config namespace: self labels: k8s-app: fluent-bit data: filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude On [FILTER] Name nest Match kube.* Operation lift Nested_under log_processed [FILTER] Name record_modifier Match * Remove_key kubernetes.annotations* Remove_key kubernetes.labels* Remove_key kubernetes.docker_id Remove_key kubernetes.labels.controller-revision-hash.keyword Remove_key stream Remove_key kubernetes.container_image Remove_key kubernetes.annotations.cni_projectcalico_org/containerID

fluent-bit.conf: | [SERVICE] Flush 1 Log_Level debug Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020

@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-elasticsearch.conf

input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/.log Exclude_Path /var/log/containers/kubesphere-monitoring-system.log,/var/log/containers/kube-system.log,/var/log/containers/kubesphere-system.log,/var/log/containers/kubesphere-devops-system.log,/var/log/containers/kubesphere-devops-worker.log,/var/log/containers/-promtail-.log,/var/log/containers/fluent-bit-.log Parser log_biz_parser DB /var/log/fluent-bit/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 # Multiline On output-elasticsearch.conf: | [OUTPUT] Name es Match kube.* Host elasticsearch Port 9200 Suppress_Type_Name On Logstash_Format On Replace_Dots On Retry_Limit False parsers.conf: | [PARSER] Name apache Format regex Regex ^(?[^ ]) [^ ] (?[^ ]) [(?[^]])] "(?\S+)(?: +(?[^"]?)(?: +\S)?)?" (?[^ ]) (?[^ ])(?: "(?[^"])" "(?[^"])")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   apache_error
    Format regex
    Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   json
    Format json
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[PARSER]
    Name        syslog
    Format      regex
    Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
    Time_Key    time
    Time_Format %b %d %H:%M:%S

[PARSER]
    Name        log_biz_parser
    Format      regex
    Regex       ^\s*(?<log_time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3})\s+(?<log_level>\w{4,6})\s*\[(?<thread>[.:\s\w-]+)\]\s+\[(?<logger>[.\s\w-]+)\]-\s+T:\[(?<traceId>[\s\w]*)\]\s+S:\[(?<spanId>[\s\w]*)\]--(?<rel_msg>.*)
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S

[zhangzx1996]

my Log look like this : 2024-05-14 10:02:03.272 INFO [ http-nio-80-exec-3] [pingApiClientAutoConfiguration]- T:[114013f40b6a74b6bdf423cd840196e2] S:[e2702174487bdd0f]--[Stripping] web client pass through sharding: GlobalContext(bizSharding=0000, now=Tue May 14 10:02:03 GMT+08:00 2024, extraInfo={})

[zhangzx1996]

My regex is matcheded in this website: https://rubular.com/

[patrick-stephens]

Are you sure that's the actual log content on disk? It does not match the K8S standard so my suspicion is you're talking about the application log output, this is then wrapped and handled by the kubelet into another format and this format is what you need to parse. Get the actual log file off the disk, don't assume anything else.

You cannot use your parser directly in the tail input, you should be using the multiline cri,docker ones first then a separate filter parser to parse your application log after that.

This comes up often enough I highlighted it here: https://calyptia.com/blog/kubernetes-metadata-enrichment-with-fluent-bit-with-troubleshooting-tips

I would also debug via stdout to see what you actually have plus also things like taking an actual log file to test with locally first: https://calyptia.com/blog/fluent-bit-tips-tricks

If your parser is matching then it will parse, if it is not then it won't so as it is not the parser must not be matching :)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.