Fluent-bit filter does not parse log based on regexp in official helm charts

[blackat]

Bug Report

Describe the bug The Parser filter does not parse the log key when deploying fluent-bit via official helm charts.

To Reproduce

• Rubular link if applicable: https://rubular.com/r/uvvLxVpJfvjVG6
• Example log message if applicable:

{"time"=>"2024-02-01T15:00:47.268049424Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2024-02-01 15:00:47,267 INFO  [i.u.servlet] (ServerService Thread Pool -- 34) Initializing Spring root WebApplicationContext"}

• Steps to reproduce the problem:
• The log parsing works if I run fluent-bit in a container, but when I run it via helm charts the log key is not parsed.
• What I have tried so far:
  • add new keys to the json works
  • change the log into logger and works
  • use as regex (?<message>((.|\n)*)) and I get all the log into value of the message key
• In case of helm I use the values.yaml manifest by default adding with this config

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level debug
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*_mercury-engine-*.log
        Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/re-cert-manager-*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
        Skip_Empty_Lines On
        Read_from_Head true

  filters: |
    [Filter]
        Name parser
        Parser wildfly_server
        Match kube.*
        Key_Name log
        Reserve_Data On

  outputs: |
    [OUTPUT]
        Name stdout
        Match kube.*
 
  customParsers: |
    [PARSER]
        Name        wildfly_server
        Format      regex
        Regex       /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,}) (?<message>((.|\n)*))/
        Time_Key    time
        Time_Format %Y-%m-%d %H:%M:%S
        Types       level:string

Expected behavior The log key is parsed and level, message keys are created in the json as in the Rubular expression editor.

Screenshots It seems that no parsing has been applied: [318] kube.var.log.containers.xyz-31a9c399_ddd-6f14.log: [[1706799647.268049423, {}], {"time"=>"2024-02-01T15:00:47.268049424Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2024-02-01 15:00:47,267 INFO [i.u.servlet] (ServerService Thread Pool -- 34) Initializing Spring root WebApplicationContext"}]

Your Environment

• Version used: 2.2.2
• helm.sh/chart: fluent-bit-0.43.0
• Configuration: added above
• Environment name and version (e.g. Kubernetes? What version?): Server Version: v1.27.7

Additional context I have to parse logs to separate the date, the log level and the message to be easily filtered in OpenSearch. Currently I am stuck after having done really many tries.

Other testing 1

Scenario: I have just applied a parser with a simple regex to the key time and it is correctly parsed, the date key is created (I preserve the time key). If I change the key to log it does not work, the format of the date is the same, at least should parse the first chunk. Log message:

 [[1706799585.689515220, {}], {"time"=>"2024-02-01T14:59:45.68951522Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2024-02-01 14:59:45,689 INFO  [xxx] (ServerService Thread Pool -- 34) Using ipFinder: TcpDiscoveryIpFinderAdapter [shared=true]"}]

Parser:

[PARSER]
        Name        wildfly_server
        Format      regex
        Regex       /^(?<date>\d{4}-\d{1,2}-\d{1,2})/

Filter:

[Filter]
        Name parser
        Parser wildfly_server
        Match kube.*
        Key_Name time
        Reserve_Data On
        Preserve_Key On

After filter applied (working):

[[1706799585.689515220, {}], {"date"=>"2024-02-01", "time"=>"2024-02-01T14:59:45.68951522Z", "stream"=>"stdout", "_p"=>"F", "log"=>"2024-02-01 14:59:45,689 INFO  [xxx] (ServerService Thread Pool -- 34) Using ipFinder: TcpDiscoveryIpFinderAdapter [shared=true]"}]

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.