fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

calyptia: switch to YAML for Fleet config

Open patrick-stephens opened this issue 1 year ago • 7 comments

Updates to using YAML for Fleet configuration now it is completed in 3.2 series. We have added this as a configurable option now with the default set to the current TOML to keep backwards behaviour - users can opt-in to YAML format and then we can also switch the default in the future.

fleet_config_legacy_format on == current INI/TOML format and the default fleet_config_legacy_format off == YAML format

In each case note the initial configuration will be assumed to be TOML, not changing installation and packaging for that.

Includes changes from cprofiles to resolve macOS failures, these should not cause an issue once that PR is merged: https://github.com/fluent/cprofiles/pull/7 Resolves the usage of bool type in the plugin and replaces with standard int flag approach.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [X] Example configuration file for the change
[INPUT]
    name cpu
    tag  cpu.local

    # Read interval (sec) Default: 1
    interval_sec 1

[CUSTOM]
    name calyptia
    API_Key ***
    Fleet_Name yaml-fleet
    fleet_config_legacy_format off

[OUTPUT]
    name  null 
    match *
  • [X] Debug log output from testing the change
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [debug] [input:calyptia_fleet:calyptia_fleet.2] parsing configuration timestamp from path: 1733416866.yaml
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [ info] [input:calyptia_fleet:calyptia_fleet.2] loading configuration from /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/new.yaml.
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [ info] [input:calyptia_fleet:calyptia_fleet.2] changing to config dir: /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866
Dec 09 14:06:53 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:53] [ info] [input:calyptia_fleet:calyptia_fleet.2] changing to config dir: /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866
...
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [custom:calyptia:calyptia.0] read UUID (334583af-cb56-4a5f-bdfa-8b4ab3f5896b) from file: /tmp/calyptia-fleet/machine-id.conf
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [custom:calyptia:calyptia.0] custom initialized!
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [simd    ] disabled
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [cmetrics] version=0.9.9
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [ctraces ] version=0.5.7
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:dummy:dummy.0] initializing
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [dummy:dummy.0] created event channels: read=20 write=21
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:fluentbit_metrics:fluentbit_metrics.1] initializing
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:fluentbit_metrics:fluentbit_metrics.1] storage_strategy='memory' (memory only)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [fluentbit_metrics:fluentbit_metrics.1] created event channels: read=22 write=25
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] initializing
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] storage_strategy='memory' (memory only)
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [calyptia_fleet:calyptia_fleet.2] created event channels: read=26 write=27
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] initializing calyptia fleet input.
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [input:calyptia_fleet:calyptia_fleet.2] initial collector interval: sec=-1 nsec=-1
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] invalid interval settings, using defaults: sec=15 nsec=0
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [input:calyptia_fleet:calyptia_fleet.2] we are already using a configuration file: /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/new.yaml
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [input:calyptia_fleet:calyptia_fleet.2] parsing configuration timestamp from path: 1733416866.yaml
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [input:calyptia_fleet:calyptia_fleet.2] fleet collector initialized with interval: 15 sec 0 nsec
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [stdout:stdout.0] created event channels: read=28 write=29
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [calyptia:calyptia.1] created event channels: read=37 write=38
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [output:calyptia:calyptia.1] machine_id=dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [ info] [output:stdout:stdout.0] worker #0 started
Dec 09 14:06:59 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:06:59] [debug] [http_client] not using http_proxy for header
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [ info] [output:calyptia:calyptia.1] connected to Calyptia, agent_id='100ddda4-d2e1-4edb-8a24-a442fa253219'
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [ info] [output:calyptia:calyptia.1] agent registration successful
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule dummy.0:stdout.0
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule fluentbit_metrics.1:stdout.0
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule fluentbit_metrics.1:calyptia.1
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [debug] [router] match rule calyptia_fleet.2:stdout.0
Dec 09 14:07:00 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:00] [ info] [sp] stream processor started
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753221.654021075, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [out flush] cb_destroy coro_id=0
Dec 09 14:07:02 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:02] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753222.654082768, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [out flush] cb_destroy coro_id=1
Dec 09 14:07:03 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:03] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753223.654070928, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [out flush] cb_destroy coro_id=2
Dec 09 14:07:04 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:04] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753224.654117339, {}], {"message"=>"custom dummy"}]
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [out flush] cb_destroy coro_id=3
Dec 09 14:07:05 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:05] [debug] [task] destroy task=0x70c01da36d20 (task_id=0)
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:06] [debug] [task] created task=0x70c01da36d20 id=0 OK
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:06] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [2024/12/09 14:07:06] [debug] [task] created direct task=0x70c01da36e60 id=1 OK
Dec 09 14:07:06 pat-fleet-test-ubuntu22 fluent-bit[2412]: [0] dummy.0: [[1733753225.654339922, {}], {"message"=>"custom dummy"}]

Confirmed all fleet config files are YAML:

$ sudo find /tmp/calyptia-fleet/
/tmp/calyptia-fleet/
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/cur.yaml
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/header.yaml
/tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866.yaml
/tmp/calyptia-fleet/machine-id.conf
$ sudo cat /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/1733416866.yaml
includes: 
    - /tmp/calyptia-fleet/dc86c10731acee376c73985ff88b8ec86888e045d6af2ccf045f94b55e834be4/yaml-fleet/header.yaml
pipeline:
    inputs:
        - name: dummy
          dummy: '{"message": "custom dummy"}'
    outputs:
        - name: stdout
          match: '*'
  • [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Run local packaging test showing all targets (including any new ones) build.
  • [x] Set ok-package-test label to test for all targets (requires maintainer to do).

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

patrick-stephens avatar Dec 09 '24 11:12 patrick-stephens

Investigating this leak:

==2877== 
==2877== HEAP SUMMARY:
==2877==     in use at exit: 4,113 bytes in 1 blocks
==2877==   total heap usage: 42,184 allocs, 42,183 frees, 16,825,439 bytes allocated
==2877== 
==2877== Thread 1:
==2877== 4,113 bytes in 1 blocks are definitely lost in loss record 1 of 1
==2877==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2877==    by 0x288562: flb_malloc (flb_mem.h:80)
==2877==    by 0x288562: sds_alloc (flb_sds.c:41)
==2877==    by 0x288562: flb_sds_create_size (flb_sds.c:93)
==2877==    by 0x33C615: generate_base_fleet_directory (in_calyptia_fleet.c:234)
==2877==    by 0x33C615: fleet_config_filename (in_calyptia_fleet.c:257)
==2877==    by 0x33E439: time_fleet_config_filename (in_calyptia_fleet.c:274)
==2877==    by 0x33E439: get_calyptia_fleet_config (in_calyptia_fleet.c:1812)
==2877==    by 0x33E439: in_calyptia_fleet_collect (in_calyptia_fleet.c:1854)
==2877==    by 0x7DA1EA: co_init (amd64.c:117)
==2877== 
==2877== LEAK SUMMARY:
==2877==    definitely lost: 4,113 bytes in 1 blocks
==2877==    indirectly lost: 0 bytes in 0 blocks
==2877==      possibly lost: 0 bytes in 0 blocks
==2877==    still reachable: 0 bytes in 0 blocks
==2877==         suppressed: 0 bytes in 0 blocks

Looks to be an existing leak with success return, only happens once though.

patrick-stephens avatar Dec 09 '24 14:12 patrick-stephens

Verified that we can use TOML by default if undefined, also verified we can switch from YAML to TOML even with existing YAML config.

patrick-stephens avatar Dec 11 '24 15:12 patrick-stephens

Testing on Windows shows an issue on reloading the first time with the new config so investigating.

patrick-stephens avatar Dec 12 '24 14:12 patrick-stephens

Packaging failures are related to Github runner update: https://github.blog/changelog/2024-11-05-notice-of-breaking-changes-for-github-actions/

https://github.com/fluent/fluent-bit/issues/9739

patrick-stephens avatar Dec 17 '24 10:12 patrick-stephens

YAML config has a case-sensitivity issue as well: https://github.com/fluent/fluent-bit/issues/9743 This was triggering failures in the test as the default set up used Name instead of name.

patrick-stephens avatar Dec 17 '24 17:12 patrick-stephens

Ah rebase messed up, let me fix it up...

patrick-stephens avatar Dec 19 '24 16:12 patrick-stephens

@patrick-stephens can you squash into smaller amount of commits?

niedbalski avatar Dec 19 '24 17:12 niedbalski

Merged in rather than rebased instead.

patrick-stephens avatar Jan 06 '25 10:01 patrick-stephens

@patrick-stephens can you squash into smaller amount of commits?

Done.

patrick-stephens avatar Jan 06 '25 10:01 patrick-stephens