fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

filter_type_converter: support map_key

Open nokute78 opened this issue 1 year ago • 4 comments

Fixes #8193

This patch is to support map_key for filter_type_converter. It is to convert from map to json string. e.g.

"map_data":{"key":"value", "id":123}

to

"map_str": "{"key":"value","id":123}"
Key Description
map_key This parameter is for string source. json_string is allowed as dst_data_type.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [X] Example configuration file for the change
  • [X] Debug log output from testing the change
  • [X] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [ ] Backport to latest stable release.

Configuration

[INPUT]
    Name dummy
    Dummy {"map_data":{"key":"value", "id":123}}

[FILTER]
    Name type_converter
    Match *
    Map_key map_data map_str json_string

[OUTPUT]
    name stdout
    match *

Debug/Valgrind output

$ valgrind --leak-check=full bin/fluent-bit -c a.conf 
==69763== Memcheck, a memory error detector
==69763== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==69763== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==69763== Command: bin/fluent-bit -c a.conf
==69763== 
Fluent Bit v2.2.1
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/18 10:45:29] [ info] [fluent bit] version=2.2.1, commit=281aea0a10, pid=69763
[2023/11/18 10:45:29] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/18 10:45:29] [ info] [cmetrics] version=0.6.4
[2023/11/18 10:45:29] [ info] [ctraces ] version=0.3.1
[2023/11/18 10:45:29] [ info] [input:dummy:dummy.0] initializing
[2023/11/18 10:45:29] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/11/18 10:45:29] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/18 10:45:29] [ info] [sp] stream processor started
[0] dummy.0: [[1700271930.009027474, {}], {"map_data"=>{"key"=>"value", "id"=>123}, "map_str"=>"{"key":"value","id":123}"}]
^C[2023/11/18 10:45:31] [engine] caught signal (SIGINT)
[2023/11/18 10:45:31] [ warn] [engine] service will shutdown in max 5 seconds
[0] dummy.0: [[1700271931.015915198, {}], {"map_data"=>{"key"=>"value", "id"=>123}, "map_str"=>"{"key":"value","id":123}"}]
[2023/11/18 10:45:31] [ info] [input] pausing dummy.0
[2023/11/18 10:45:31] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/18 10:45:31] [ info] [input] pausing dummy.0
[2023/11/18 10:45:32] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/18 10:45:32] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==69763== 
==69763== HEAP SUMMARY:
==69763==     in use at exit: 0 bytes in 0 blocks
==69763==   total heap usage: 1,797 allocs, 1,797 frees, 1,191,670 bytes allocated
==69763== 
==69763== All heap blocks were freed -- no leaks are possible
==69763== 
==69763== For lists of detected and suppressed errors, rerun with: -s
==69763== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

nokute78 avatar Nov 18 '23 01:11 nokute78

@nokute78 Hi! I don't quite understand where I can find the artifacts, could you give me a link to the image, I want to test, thanks!

kinseii avatar Feb 13 '24 07:02 kinseii

Any updates on when this might be published? This would be extremely useful.

jpage-godaddy avatar May 09 '24 20:05 jpage-godaddy

I also need this function

klzsysy avatar May 20 '24 15:05 klzsysy