config: there is no support for list properties in the yaml format

[pwhelan]

Bug Report

The new YAML format is unable to support list properties. It is impossible to configure multiple values for a single property for properties that support it, such as the filter checklist property record which is used to set the value for several record properties.

To Reproduce

Instead it gives this error:

Fluent Bit v1.9.4
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/05/12 12:04:53] [error] [config] YAML error found in file "test.yaml", line 17, column 18: unexpected event 7 in state 21.
[2022/05/12 12:04:53] [error] configuration file contains errors, aborting.

• Steps to reproduce the problem:

Use this example configuration:

---
pipeline:
    inputs:
        - dummy:
              dummy: '{"remote_addr": "127.0.0.1", "message": "FOOBAR"}'
              tag: remote
        - dummy:
              dummy: '{"remote_addr": "7.7.7.7", "message": "MALICIOUS"}'
              tag: remote
    filters:
        - checklist:
              name: checklist
              match: remote
              file: ip_list.txt
              lookup_key: $remote_addr
              record:
                  - malicious true
                  - tagged true
    outputs:
        - stdout:
              match: remote

Create the following ip_list.txt file:

7.7.7.7

Then invoke fluent-bit:

$ fluent-bit -c test.yaml -f 1 -v
Fluent Bit v1.9.4
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/05/12 12:04:53] [error] [config] YAML error found in file "test.yaml", line 17, column 18: unexpected event 7 in state 21.
[2022/05/12 12:04:53] [error] configuration file contains errors, aborting.

Expected behavior

It should reproduce the following records:

[0] remote: [1652371169.597327537, {"remote_addr"=>"127.0.0.1", "message"=>"FOOBAR"}]
[0] remote: [1652371169.597356592, {"remote_addr"=>"7.7.7.7", "message"=>"MALICIOUS", "malicious"=>true, "tagged" =>true}]

Your Environment

• Version used: 1.9.4
• Configuration: shown above
• Environment name and version: Arch Linux 20220512
• Server type and version: Arch Linux 20220512
• Operating System and version: Arch Linux 20220512
• Filters and plugins: dummy, checklist and stdout

[nokute78]

Below yaml works.

pipeline:
    inputs:
        - dummy:
              dummy: '{"remote_addr": "127.0.0.1", "message": "FOOBAR"}'
              tag: remote
        - dummy:
              dummy: '{"remote_addr": "7.7.7.7", "message": "MALICIOUS"}'
              tag: remote
    filters:
        - checklist:
              name: checklist
              match: remote
              file: ip_list.txt
              lookup_key: $remote_addr
              record:
                  malicious true
                  tagged true
    outputs:
        - stdout:
              match: remote

diff is

--- org.yaml	2022-07-18 08:17:38.864404916 +0900
+++ a.yaml	2022-07-18 08:17:06.775738996 +0900
@@ -13,8 +13,8 @@
               file: ip_list.txt
               lookup_key: $remote_addr
               record:
-                  - malicious true
-                  - tagged true
+                  malicious true
+                  tagged true
     outputs:
         - stdout:
               match: remote

output:

$ ../bin/fluent-bit -c a.yaml 
Fluent Bit v2.0.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/07/18 08:19:21] [ info] [fluent bit] version=2.0.0, commit=47cdf2c385, pid=3956
[2022/07/18 08:19:21] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/07/18 08:19:21] [ info] [cmetrics] version=0.3.5
[2022/07/18 08:19:21] [ info] [filter:checklist:checklist.0] load file elapsed time (sec.ns): 0.52599
[2022/07/18 08:19:21] [ info] [sp] stream processor started
[2022/07/18 08:19:21] [ info] [output:stdout:stdout.0] worker #0 started
[0] remote: [1658099962.344599215, {"remote_addr"=>"127.0.0.1", "message"=>"FOOBAR"}]
[0] remote: [1658099962.344678776, {"remote_addr"=>"7.7.7.7", "message"=>"MALICIOUS", "malicious"=>"tagged true"}]
^C[2022/07/18 08:19:23] [engine] caught signal (SIGINT)

[martian4202]

Just find something interesting

add: '_service_name_ ${service_name}',
'add ': '_level_ error',
' add': '_msg_: catch stderr',

this just works!

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[braunsonm]

Bump

[edsiper]

@pwhelan per my understanding after 2.0 changes this is not longer an issue, can you confirm pls ?

[braunsonm]

@edsiper This is still broken. YAML:

inputs:
- systemd:
    systemd_filter:
    - "_SYSTEMD_UNIT=example.service"
    - "_SYSTEMD_UNIT=example2.service"
    tag: "systemd"

Logs:

YAML error found in file ..., unexpected event 7 in state 23

[braunsonm]

Correction, this is fixed. It looks like the configuration format of YAML has changed and not been documented. Issue here: https://github.com/fluent/fluent-bit/issues/6408#issuecomment-1315784647

This works in the case I posted above but does not work in others, for instance:

service:
  parsers_file: ["parsers.conf", "another-parsers.conf"]

Still does not work

[braunsonm]

Just writing into confirm this still does not work on parsers_file as of 2.0.10

[9numbernine9]

Started running into this issue as well. We upgraded to version 0.28.0 of the Helm chart that was released a couple days ago, which now uses fluent-bit 2.1.2 by default, and multiple systemd_filter entries as a YAML list breaks the configuration file. 😢

A fix version would be greatly appreciated!

[slyt]

A very similar issue exists when configuring the modify filter with YAML.

Classic syntax works like:

[FILTER]
    Name modify
    Match kube.*
    Add label_1 my_label_value_1
    Add label_2 my_label_value_2

In YAML, this does not work:

pipeline:
  filters:
    - name: modify
      match: kube.*
      add: 
        - 'label_1 my_label_value_1'
        - 'label_2 my_label_value_2'

However, using invalid YAML syntax does work (confusingly):

pipeline:
  filters:
    - name: modify
      match: kube.*
      add: 'label_1 my_label_value_1'
      add: 'label_2 my_label_value_2'

It is strange that fluent-bit accepts the invalid YAML syntax. Additionally, this invalid YAML syntax breaks our CI/CD pipeline at the YAML linting step, so it will require a special exception rule until fixed.

[edsiper]

fixed.