out_influxdb: add dynamic tags list feature to influxdb output plugin

[PetarBersic]

InfluxDB uses tags for indexing. Before this commit the only way to set tag keys was with Tag_Keys config option. This commit adds dynamic tags by reading the Tags_List_Key string array from each log record. This feature may be turned on with Tags_List_Enabled config option.

Signed-off-by: PetarBersic [email protected]

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [x] Example configuration file for the change
• [x] Debug log output from testing the change

• [x] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• [x] Documentation required for this feature

Doc PR #541

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Example configuration

[INPUT]
    Name              dummy
    Dummy             {"msg": "Transfer completed", "level": "info", "ID": "1234", "businessObjectID": "qwerty", "status": "OK", "tags": ["ID", "businessObjectID"]}

[OUTPUT]
    Name          influxdb
    Match         *
    Host          127.0.0.1
    Port          8086
    Bucket        logs
    Org           syntio
    Sequence_Tag  _seq
    HTTP_Token    zQI_8XD4neKTRc92w8aPPjPzX_gaS5Q6zL6iZiA6P4vgAOOEbnq_JX9Mkp0U9vRjx05Os53lZOMoywsZ1zOTVw==
    Tags_List_Enabled True
    Tags_List_Key tags
    Tag_Keys level status

Debug Log Output

Fluent Bit v1.8.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/06/01 14:01:52] [ info] Configuration:
[2021/06/01 14:01:52] [ info]  flush time     | 1.000000 seconds
[2021/06/01 14:01:52] [ info]  grace          | 5 seconds
[2021/06/01 14:01:52] [ info]  daemon         | 0
[2021/06/01 14:01:52] [ info] ___________
[2021/06/01 14:01:52] [ info]  inputs:
[2021/06/01 14:01:52] [ info]      dummy
[2021/06/01 14:01:52] [ info] ___________
[2021/06/01 14:01:52] [ info]  filters:
[2021/06/01 14:01:52] [ info] ___________
[2021/06/01 14:01:52] [ info]  outputs:
[2021/06/01 14:01:52] [ info]      influxdb.0
[2021/06/01 14:01:52] [ info] ___________
[2021/06/01 14:01:52] [ info]  collectors:
[2021/06/01 14:01:52] [ info] [engine] started (pid=222628)
[2021/06/01 14:01:52] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/06/01 14:01:52] [debug] [storage] [cio stream] new stream registered: dummy.0
[2021/06/01 14:01:52] [ info] [storage] version=1.1.1, initializing...
[2021/06/01 14:01:52] [ info] [storage] in-memory
[2021/06/01 14:01:52] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/06/01 14:01:52] [debug] [influxdb:influxdb.0] created event channels: read=18 write=19
[2021/06/01 14:01:52] [debug] [output:influxdb:influxdb.0] host=127.0.0.1 port=8086
[2021/06/01 14:01:52] [debug] [router] match rule dummy.0:influxdb.0
[2021/06/01 14:01:52] [ info] [sp] stream processor started
[2021/06/01 14:01:54] [debug] [task] created task=0x7ffff0006ea0 id=0 OK
[2021/06/01 14:02:03] [debug] [task] created task=0x7ffff00237a0 id=1 OK
[2021/06/01 14:02:03] [debug] [http_client] not using http_proxy for header
[2021/06/01 14:02:03] [debug] [http_client] header=POST /api/v2/write?org=syntio&bucket=logs&precision=ns HTTP/1.1

Host: 127.0.0.1:8086

Content-Length: 112

User-Agent: Fluent-Bit

Authorization: Token zQI_8XD4neKTRc92w8aPPjPzX_gaS5Q6zL6iZiA6P4vgAOOEbnq_JX9Mkp0U9vRjx05Os53lZOMoywsZ1zOTVw==

Valgrind Output

$ valgrind --leak-check=full bin/fluent-bit -c influxdb_example.conf
==292454== Memcheck, a memory error detector
==292454== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==292454== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info
==292454== Command: bin/fluent-bit -c influxdb_example.conf
==292454==
Fluent Bit v1.8.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/06/01 14:35:10] [ info] Configuration:
[2021/06/01 14:35:10] [ info]  flush time     | 1.000000 seconds
[2021/06/01 14:35:10] [ info]  grace          | 5 seconds
[2021/06/01 14:35:10] [ info]  daemon         | 0
[2021/06/01 14:35:10] [ info] ___________
[2021/06/01 14:35:10] [ info]  inputs:
[2021/06/01 14:35:10] [ info]      dummy
[2021/06/01 14:35:10] [ info] ___________
[2021/06/01 14:35:10] [ info]  filters:
[2021/06/01 14:35:10] [ info] ___________
[2021/06/01 14:35:10] [ info]  outputs:
[2021/06/01 14:35:10] [ info]      influxdb.0
[2021/06/01 14:35:10] [ info] ___________
[2021/06/01 14:35:10] [ info]  collectors:
[2021/06/01 14:35:10] [ info] [engine] started (pid=292454)
[2021/06/01 14:35:10] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/06/01 14:35:10] [debug] [storage] [cio stream] new stream registered: dummy.0
[2021/06/01 14:35:10] [ info] [storage] version=1.1.1, initializing...
[2021/06/01 14:35:10] [ info] [storage] in-memory
[2021/06/01 14:35:10] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/06/01 14:35:10] [debug] [influxdb:influxdb.0] created event channels: read=18 write=19
[2021/06/01 14:35:10] [debug] [output:influxdb:influxdb.0] host=127.0.0.1 port=8086
[2021/06/01 14:35:10] [debug] [router] match rule dummy.0:influxdb.0
[2021/06/01 14:35:10] [ info] [sp] stream processor started
[2021/06/01 14:35:12] [debug] [task] created task=0x4c509d0 id=0 OK
[2021/06/01 14:35:12] [debug] [http_client] not using http_proxy for header
[2021/06/01 14:35:12] [debug] [http_client] header=POST /api/v2/write?org=syntio&bucket=logs&precision=ns HTTP/1.1
Host: 127.0.0.1:8086
Content-Length: 112
User-Agent: Fluent-Bit
Authorization: Token zQI_8XD4neKTRc92w8aPPjPzX_gaS5Q6zL6iZiA6P4vgAOOEbnq_JX9Mkp0U9vRjx05Os53lZOMoywsZ1zOTVw==


[2021/06/01 14:35:12] [debug] [output:influxdb:influxdb.0] http_do=0 OK
[2021/06/01 14:35:12] [debug] [out coro] cb_destroy coro_id=0
[2021/06/01 14:35:12] [debug] [task] destroy task=0x4c509d0 (task_id=0)
^C[2021/06/01 14:35:12] [engine] caught signal (SIGINT)
[2021/06/01 14:35:12] [debug] [task] created task=0x4cc3560 id=0 OK
[2021/06/01 14:35:12] [ warn] [engine] service will stop in 5 seconds
[2021/06/01 14:35:12] [debug] [http_client] not using http_proxy for header
[2021/06/01 14:35:12] [debug] [http_client] header=POST /api/v2/write?org=syntio&bucket=logs&precision=ns HTTP/1.1
Host: 127.0.0.1:8086
Content-Length: 112
User-Agent: Fluent-Bit
Authorization: Token zQI_8XD4neKTRc92w8aPPjPzX_gaS5Q6zL6iZiA6P4vgAOOEbnq_JX9Mkp0U9vRjx05Os53lZOMoywsZ1zOTVw==


[2021/06/01 14:35:12] [debug] [output:influxdb:influxdb.0] http_do=0 OK
[2021/06/01 14:35:12] [debug] [out coro] cb_destroy coro_id=1
[2021/06/01 14:35:12] [debug] [task] destroy task=0x4cc3560 (task_id=0)
[2021/06/01 14:35:13] [debug] [input chunk] dummy.0 is paused, cannot append records
[2021/06/01 14:35:14] [debug] [input chunk] dummy.0 is paused, cannot append records
[2021/06/01 14:35:15] [debug] [input chunk] dummy.0 is paused, cannot append records
[2021/06/01 14:35:16] [debug] [input chunk] dummy.0 is paused, cannot append records
[2021/06/01 14:35:17] [debug] [input chunk] dummy.0 is paused, cannot append records
[2021/06/01 14:35:17] [ info] [engine] service stopped
==292454==
==292454== HEAP SUMMARY:
==292454==     in use at exit: 0 bytes in 0 blocks
==292454==   total heap usage: 576 allocs, 576 frees, 1,244,949 bytes allocated
==292454==
==292454== All heap blocks were freed -- no leaks are possible
==292454==
==292454== For lists of detected and suppressed errors, rerun with: -s
==292454== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[edsiper]

bump

[PetarBersic]

Fixed memory leak in the fluent:master branch and resolved merge conflicts.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.