out_syslog: handle multivalued structured data

[bplessis]

When generating a syslog SD-ELEMENT it is permitted to repeat a SD-PARAM multiple times: https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

This is used by loggly to pass tags alongside syslog messages for example https://documentation.solarwinds.com/en/success_center/loggly/content/admin/tags.htm

This proposal convert array-typed object into multiple SD-PARAM elements to be able to satisfy this need.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [x] Example configuration file for the change
• [x] Debug log output from testing the change

• [x] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [x] Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[bplessis]

Example configuration file for the change:

[INPUT]
  Name  dummy
  Dummy {"hostname":"test.example.org", "appname": "my-application", "procid": "123", "msgid": "ID01", "log": "test log message", "uls@0": {"tag":[ "tag1", "tag2", "tag3", "tag4"], "key": "value" } }

[OUTPUT]
    name                 syslog
    match                *
    host                 127.0.0.1
    port                 5140
    mode                 udp
    syslog_format        rfc5424
    syslog_maxsize       2048
    syslog_hostname_key  hostname
    syslog_appname_key   appname
    syslog_procid_key    procid
    syslog_msgid_key     msgid    
    syslog_sd_key        uls@0
    syslog_message_key   log

[bplessis]

Debug run

Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

       \ || /
       = ** =
        /__\ 
       /____\
      /______\
     /________\
    /__________\
   /____________\
  /__Fluent Bit__\
 /__|__v2.2.1_|___\
/___\_________/____\
        |||
        |||
       =====

[2024/01/06 23:35:23] [ info] Configuration:
[2024/01/06 23:35:23] [ info]  flush time     | 1.000000 seconds
[2024/01/06 23:35:23] [ info]  grace          | 5 seconds
[2024/01/06 23:35:23] [ info]  daemon         | 0
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  inputs:
[2024/01/06 23:35:23] [ info]      dummy
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  filters:
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  outputs:
[2024/01/06 23:35:23] [ info]      syslog.0
[2024/01/06 23:35:23] [ info] ___________
[2024/01/06 23:35:23] [ info]  collectors:
[2024/01/06 23:35:23] [ info] [fluent bit] version=2.2.2, commit=f2692ed152, pid=1158178
[2024/01/06 23:35:23] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/01/06 23:35:23] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/01/06 23:35:23] [ info] [cmetrics] version=0.6.6
[2024/01/06 23:35:23] [ info] [ctraces ] version=0.4.0
[2024/01/06 23:35:23] [ info] [input:dummy:dummy.0] initializing
[2024/01/06 23:35:23] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/01/06 23:35:23] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/01/06 23:35:23] [debug] [syslog:syslog.0] created event channels: read=23 write=24
[2024/01/06 23:35:23] [ info] [output:syslog:syslog.0] setup done for 127.0.0.1:5140 (TLS=off)
[2024/01/06 23:35:23] [ info] [sp] stream processor started
[2024/01/06 23:35:24] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:25] [trace] [task 0x7f9ac4018ef0] created (id=0)
[2024/01/06 23:35:25] [debug] [task] created task=0x7f9ac4018ef0 id=0 OK
[2024/01/06 23:35:25] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:25] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:25] [debug] [out flush] cb_destroy coro_id=0
[2024/01/06 23:35:25] [trace] [coro] destroy coroutine=0x7f9ac4019010 data=0x7f9ac4019030
[2024/01/06 23:35:25] [debug] [task] destroy task=0x7f9ac4018ef0 (task_id=0)
[2024/01/06 23:35:26] [trace] [task 0x7f9ac4019300] created (id=0)
[2024/01/06 23:35:26] [debug] [task] created task=0x7f9ac4019300 id=0 OK
[2024/01/06 23:35:26] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:26] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:26] [debug] [out flush] cb_destroy coro_id=1
[2024/01/06 23:35:26] [trace] [coro] destroy coroutine=0x7f9ac4019420 data=0x7f9ac4019440
[2024/01/06 23:35:26] [debug] [task] destroy task=0x7f9ac4019300 (task_id=0)
[2024/01/06 23:35:27] [trace] [task 0x7f9ac40195f0] created (id=0)
[2024/01/06 23:35:27] [debug] [task] created task=0x7f9ac40195f0 id=0 OK
[2024/01/06 23:35:27] [debug] [input chunk] update output instances with new chunk size diff=156, records=1, input=dummy.0
[2024/01/06 23:35:27] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:27] [debug] [out flush] cb_destroy coro_id=2
[2024/01/06 23:35:27] [trace] [coro] destroy coroutine=0x7f9ac40196e0 data=0x7f9ac4019700
[2024/01/06 23:35:27] [debug] [task] destroy task=0x7f9ac40195f0 (task_id=0)
^C[2024/01/06 23:35:28] [engine] caught signal (SIGINT)
[2024/01/06 23:35:28] [trace] [engine] flush enqueued data
[2024/01/06 23:35:28] [trace] [task 0x7f9ac4019830] created (id=0)
[2024/01/06 23:35:28] [debug] [task] created task=0x7f9ac4019830 id=0 OK
[2024/01/06 23:35:28] [ warn] [engine] service will shutdown in max 5 seconds
[2024/01/06 23:35:28] [ info] [input] pausing dummy.0
[2024/01/06 23:35:28] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/01/06 23:35:28] [debug] [out flush] cb_destroy coro_id=3
[2024/01/06 23:35:28] [trace] [coro] destroy coroutine=0x7f9ac40191b0 data=0x7f9ac40191d0
[2024/01/06 23:35:28] [debug] [task] destroy task=0x7f9ac4019830 (task_id=0)
[2024/01/06 23:35:28] [ info] [engine] service has stopped (0 pending tasks)
[2024/01/06 23:35:28] [ info] [input] pausing dummy.0

[bplessis]

Valgrind run

==1157749== Memcheck, a memory error detector
==1157749== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==1157749== Using Valgrind-3.20.0 and LibVEX; rerun with -h for copyright info
==1157749== Command: bin/fluent-bit -c bin/test.conf
==1157749== 
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

       \ || /
       = ** =
        /__\ 
       /____\
      /______\
     /________\
    /__________\
   /____________\
  /__Fluent Bit__\
 /__|__v2.2.1_|___\
/___\_________/____\
        |||
        |||
       =====

[2024/01/06 23:22:52] [ info] [fluent bit] version=2.2.2, commit=f2692ed152, pid=1157749
[2024/01/06 23:22:52] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/01/06 23:22:52] [ info] [cmetrics] version=0.6.6
[2024/01/06 23:22:52] [ info] [ctraces ] version=0.4.0
[2024/01/06 23:22:52] [ info] [input:dummy:dummy.0] initializing
[2024/01/06 23:22:52] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/01/06 23:22:52] [ info] [output:syslog:syslog.0] setup done for 127.0.0.1:5140 (TLS=off)
[2024/01/06 23:22:52] [ info] [sp] stream processor started
[2024/01/06 23:22:54] [ warn] [engine] failed to flush chunk '1157749-1704579773.514668660.flb', retry in 8 seconds: task_id=0, input=dummy.0 > output=syslog.0 (out_id=0)
[2024/01/06 23:22:56] [ warn] [engine] failed to flush chunk '1157749-1704579775.501811336.flb', retry in 7 seconds: task_id=1, input=dummy.0 > output=syslog.0 (out_id=0)
[2024/01/06 23:23:02] [ info] [engine] flush chunk '1157749-1704579773.514668660.flb' succeeded at retry 1: task_id=0, input=dummy.0 > output=syslog.0 (out_id=0)
[2024/01/06 23:23:03] [ info] [engine] flush chunk '1157749-1704579775.501811336.flb' succeeded at retry 1: task_id=1, input=dummy.0 > output=syslog.0 (out_id=0)
^C[2024/01/06 23:23:07] [engine] caught signal (SIGINT)
[2024/01/06 23:23:07] [ warn] [engine] service will shutdown in max 5 seconds
[2024/01/06 23:23:07] [ info] [input] pausing dummy.0
[2024/01/06 23:23:08] [ info] [engine] service has stopped (0 pending tasks)
[2024/01/06 23:23:08] [ info] [input] pausing dummy.0
==1157749== 
==1157749== HEAP SUMMARY:
==1157749==     in use at exit: 0 bytes in 0 blocks
==1157749==   total heap usage: 2,299 allocs, 2,299 frees, 6,031,600 bytes allocated
==1157749== 
==1157749== All heap blocks were freed -- no leaks are possible
==1157749== 
==1157749== For lists of detected and suppressed errors, rerun with: -s
==1157749== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[bplessis-swi]

Do i need to do something more to help this ? @edsiper @leonardo-albertovich @fujimotos @koleini ?

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[bplessis-swi]

how can we proceed now ? @edsiper @leonardo-albertovich @fujimotos @koleini ?

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[bplessis-swi]

No sure what is missing here .. how can we proceed now ? @edsiper @leonardo-albertovich @fujimotos @koleini ?

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[eschabell]

Docs for this PR available here: https://github.com/fluent/fluent-bit-docs/pull/1292

[eschabell]

@edsiper can someone review this PR?