fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

parser: add Time_System_Timezone

Open braydonk opened this issue 2 years ago • 4 comments

Added a new config option for parsers called time_system_timezone which will fall back to mktime for tm to time conversion which will force the usage of the system timezone.

Fixes #593

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Example configuration file for the change

a.txt

{"message": "test", "time": "2023-11-11:13:00:00"}
{"message": "test2", "time": "2023-10-17:00:00:00"}

parser.conf

[PARSER]
    Name        no_timezone
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%d:%H:%M:%S
    Time_System_Timezone true

config.conf

[SERVICE]
  Log_Level debug

[INPUT]
    Name           tail
    Tag            test
    Exit_On_Eof    true
    Read_From_Head true
    Path           a.txt
    Key            log

[FILTER]
    Name     parser
    Match    *
    Parser   no_timezone 
    Key_Name log

[OUTPUT]
    Name   stdout
    Match  *
  • [x] Debug log output from testing the change

Linux

braydonk@braydonk:~/Git/fluent-bit$ ./build/bin/fluent-bit --conf ~/Documents/fluent-bit-configs/config.conf --parser ~/Documents/fluent-bit-configs/parser.conf 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/12 19:30:17] [ info] Configuration:
[2023/11/12 19:30:17] [ info]  flush time     | 1.000000 seconds
[2023/11/12 19:30:17] [ info]  grace          | 5 seconds
[2023/11/12 19:30:17] [ info]  daemon         | 0
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  inputs:
[2023/11/12 19:30:17] [ info]      tail
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  filters:
[2023/11/12 19:30:17] [ info]      parser.0
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  outputs:
[2023/11/12 19:30:17] [ info]      stdout.0
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  collectors:
[2023/11/12 19:30:17] [ info] [fluent bit] version=2.2.0, commit=1c4c9ec2e3, pid=385488
[2023/11/12 19:30:17] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/11/12 19:30:17] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/12 19:30:17] [ info] [cmetrics] version=0.6.3
[2023/11/12 19:30:17] [ info] [ctraces ] version=0.3.1
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] initializing
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/12 19:30:17] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] inotify watch fd=27
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] scanning path a.txt
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] inode=5906182 with offset=0 appended as a.txt
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] scan_glob add(): a.txt, inode 5906182
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] 1 new files found on path 'a.txt'
[2023/11/12 19:30:17] [debug] [stdout:stdout.0] created event channels: read=29 write=30
[2023/11/12 19:30:17] [ info] [sp] stream processor started
[2023/11/12 19:30:17] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/12 19:30:17] [debug] [input chunk] update output instances with new chunk size diff=71, records=2, input=tail.0
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] [static files] processed 103b
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] inode=5906182 file=a.txt ended, stop
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] inode=5906182 file=a.txt promote to TAIL_EVENT
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5906182 watch_fd=1 name=a.txt
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/11/12 19:30:17] [debug] [task] created task=0x7f0664038ea0 id=0 OK
[2023/11/12 19:30:17] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/11/12 19:30:17] [ warn] [engine] service will shutdown in max 5 seconds
[2023/11/12 19:30:17] [ info] [input] pausing tail.0
[0] test: [[1699725600.000000000, {}], {"message"=>"test"}]
[1] test: [[1697515200.000000000, {}], {"message"=>"test2"}]
[2023/11/12 19:30:17] [debug] [out flush] cb_destroy coro_id=0
[2023/11/12 19:30:17] [debug] [task] destroy task=0x7f0664038ea0 (task_id=0)
[2023/11/12 19:30:18] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/12 19:30:18] [ info] [input] pausing tail.0
[2023/11/12 19:30:18] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/12 19:30:18] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/11/12 19:30:18] [debug] [input:tail:tail.0] inode=5906182 removing file name a.txt
[2023/11/12 19:30:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=5906182 watch_fd=1
braydonk@braydonk:~/Git/fluent-bit$ date -d @1699725600
Sat Nov 11 01:00:00 PM EST 2023
braydonk@braydonk:~/Git/fluent-bit$ date -d @1697515200
Tue Oct 17 12:00:00 AM EDT 2023

Note that the timestamps printed from date -d can be compared with the log lines from a.txt above. They are both correctly interpreted as being the time in the timestamp at my system timezone, which is Eastern Time. Daylight savings is correctly determined for Oct 17, which occurs before the clocks go back on Nov 5. If your system is in a different timezone, running TZ=EST5EDT ./fluent-bit ... etc should produce the same result as what I got.

Windows

PS C:\Users\braydonk\Documents> C:\Users\braydonk\Git\fluent-bit\build\bin\Debug\fluent-bit.exe --config config.conf --parser parser.conf
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/12 19:55:28] [ info] Configuration:
[2023/11/12 19:55:28] [ info]  flush time     | 1.000000 seconds
[2023/11/12 19:55:28] [ info]  grace          | 5 seconds
[2023/11/12 19:55:28] [ info]  daemon         | 0
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  inputs:
[2023/11/12 19:55:28] [ info]      tail
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  filters:
[2023/11/12 19:55:28] [ info]      parser.0
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  outputs:
[2023/11/12 19:55:28] [ info]      stdout.0
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  collectors:
[2023/11/12 19:55:28] [ info] [fluent bit] version=2.2.0, commit=640762fb33, pid=4328
[2023/11/12 19:55:28] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2023/11/12 19:55:28] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/12 19:55:28] [ info] [cmetrics] version=0.6.3
[2023/11/12 19:55:28] [ info] [ctraces ] version=0.3.1
[2023/11/12 19:55:28] [ info] [input:tail:tail.0] initializing
[2023/11/12 19:55:28] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/12 19:55:28] [debug] [tail:tail.0] created event channels: read=820 write=824
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] inode=53198770598527222 with offset=0 appended as C:\Users\braydonk\Documents\a.txt
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] 1 new files found on path 'a.txt'
[2023/11/12 19:55:28] [debug] [stdout:stdout.0] created event channels: read=848 write=852
[2023/11/12 19:55:28] [ info] [sp] stream processor started
[2023/11/12 19:55:28] [debug] [input chunk] update output instances with new chunk size diff=71, records=2, input=tail.0
[2023/11/12 19:55:28] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] [static files] processed 105b
[0] test: [[1699725600.000000000, [2023/11/12 19:55:28] [ info] [input:tail:tail.0] inode=53198770598527222 file=C:\Users\braydonk\Documents\a.txt ended, stop
{}], [2023/11/12 19:55:28] [debug] [input:tail:tail.0] inode=53198770598527222 file=C:\Users\braydonk\Documents\a.txt promote to TAIL_EVENT
{[2023/11/12 19:55:28] [debug] [input:tail:tail.0] [static files] processed 0b, done
"[2023/11/12 19:55:28] [debug] [task] created task=000002A3795D4AA0 id=0 OK
message[2023/11/12 19:55:28] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
"[2023/11/12 19:55:28] [ warn] [engine] service will shutdown in max 5 seconds
=>"[2023/11/12 19:55:28] [ info] [input] pausing tail.0
test"}]
[1] test: [[1697515200.000000000, {}], {"message"=>"test2"}]
[2023/11/12 19:55:28] [debug] [out flush] cb_destroy coro_id=0
[2023/11/12 19:55:28] [debug] [task] destroy task=000002A3795D4AA0 (task_id=0)
[2023/11/12 19:55:29] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/12 19:55:29] [ info] [input] pausing tail.0
[2023/11/12 19:55:29] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/12 19:55:29] [debug] [input:tail:tail.0] inode=53198770598527222 removing file name C:\Users\braydonk\Documents\a.txt
[2023/11/12 19:55:29] [ info] [output:stdout:stdout.0] thread worker #0 stopped

With my Windows machine set to the same timezone as my Linux machine, I got the same timestamp results. (I couldn't tell you why the output from stdout is mangled like that, but it's the same on master :shrug: )

  • [x] Attached Valgrind output that shows no leaks or memory corruption was found
braydonk@braydonk:~/Git/fluent-bit$ valgrind --leak-check=full ./buildvalgrind/bin/fluent-bit --conf ~/Documents/fluent-bit-configs/config.conf --parser ~/Documents/fluent-bit-configs/parser.conf 
==399441== Memcheck, a memory error detector
==399441== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==399441== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==399441== Command: ./buildvalgrind/bin/fluent-bit --conf /home/braydonk/Documents/fluent-bit-configs/config.conf --parser /home/braydonk/Documents/fluent-bit-configs/parser.conf
==399441== 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/12 20:04:53] [ info] Configuration:
[2023/11/12 20:04:53] [ info]  flush time     | 1.000000 seconds
[2023/11/12 20:04:53] [ info]  grace          | 5 seconds
[2023/11/12 20:04:53] [ info]  daemon         | 0
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  inputs:
[2023/11/12 20:04:53] [ info]      tail
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  filters:
[2023/11/12 20:04:53] [ info]      parser.0
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  outputs:
[2023/11/12 20:04:53] [ info]      stdout.0
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  collectors:
[2023/11/12 20:04:53] [ info] [fluent bit] version=2.2.0, commit=6af61749af, pid=399441
[2023/11/12 20:04:53] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/11/12 20:04:53] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/12 20:04:53] [ info] [cmetrics] version=0.6.3
[2023/11/12 20:04:53] [ info] [ctraces ] version=0.3.1
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] initializing
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/12 20:04:53] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/11/12 20:04:53] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] inotify watch fd=27
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] scanning path a.txt
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] inode=5906182 with offset=0 appended as a.txt
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] scan_glob add(): a.txt, inode 5906182
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] 1 new files found on path 'a.txt'
[2023/11/12 20:04:53] [debug] [stdout:stdout.0] created event channels: read=29 write=30
[0] test: [[1699725600.000000000, {}], {"message"=>"test"}]
[1] test: [[1697515200.000000000, {}], {"message"=>"test2"}]
[2023/11/12 20:04:53] [ info] [sp] stream processor started
[2023/11/12 20:04:53] [debug] [out flush] cb_destroy coro_id=0
[2023/11/12 20:04:53] [debug] [input chunk] update output instances with new chunk size diff=71, records=2, input=tail.0
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] [static files] processed 103b
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] inode=5906182 file=a.txt ended, stop
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] inode=5906182 file=a.txt promote to TAIL_EVENT
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5906182 watch_fd=1 name=a.txt
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/11/12 20:04:53] [debug] [task] created task=0x5fd5e30 id=0 OK
[2023/11/12 20:04:53] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/11/12 20:04:53] [ warn] [engine] service will shutdown in max 5 seconds
[2023/11/12 20:04:53] [ info] [input] pausing tail.0
[2023/11/12 20:04:53] [debug] [task] destroy task=0x5fd5e30 (task_id=0)
[2023/11/12 20:04:54] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/12 20:04:54] [ info] [input] pausing tail.0
[2023/11/12 20:04:54] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/12 20:04:54] [debug] [input:tail:tail.0] inode=5906182 removing file name a.txt
[2023/11/12 20:04:54] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/11/12 20:04:54] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=5906182 watch_fd=1
==399441== 
==399441== HEAP SUMMARY:
==399441==     in use at exit: 0 bytes in 0 blocks
==399441==   total heap usage: 3,313 allocs, 3,313 frees, 983,448 bytes allocated
==399441== 
==399441== All heap blocks were freed -- no leaks are possible
==399441== 
==399441== For lists of detected and suppressed errors, rerun with: -s
==399441== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Documentation

  • [x] Documentation required for this feature

Will add when ready.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

braydonk avatar Nov 13 '23 01:11 braydonk

The MacOS test failure looks like a flake to me.

braydonk avatar Nov 13 '23 19:11 braydonk

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions[bot] avatar Feb 27 '24 01:02 github-actions[bot]

not stale

braydonk avatar Feb 27 '24 14:02 braydonk

The Windows unit test failure looks like a flake; I didn't change anything in that test with this PR.

braydonk avatar Apr 29 '24 18:04 braydonk

A good PR, can't wait for it to be validated and included in a release...

tguenneguez avatar May 29 '24 19:05 tguenneguez

Hi @braydonk, can you please add the docs PR?

lecaros avatar Jun 04 '24 14:06 lecaros

@lecaros done: https://github.com/fluent/fluent-bit-docs/pull/1388

braydonk avatar Jun 04 '24 14:06 braydonk

nginx error_log is already chomping at the bit

s5unty avatar Jun 08 '24 10:06 s5unty

@braydonk thanks for this contribution! we are ready to start the merge.

Would you please split the commits per interface ? we will need as a minimum commits for:

  • filter_kubernetes
  • parser
  • multiline
  • tests: internal: fuzzers

thank you

edsiper avatar Jun 13 '24 03:06 edsiper

I've pushed a new PR with clean commits on #9006

edsiper avatar Jun 24 '24 19:06 edsiper

merged in https://github.com/fluent/fluent-bit/pull/9006

edsiper avatar Jun 25 '24 00:06 edsiper