fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

in_syslog: handle null byte to prevent sending incomplete message into parser

Open pawpy opened this issue 5 years ago • 2 comments

To address #2741 Skip ending byte by advancing buf_parsed and have consume_bytes be left out of while loop that is collecting the log messages for the parser

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • [N/A] Debug log output from testing the change
  • [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • [N/A] Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

pawpy avatar Nov 01 '20 13:11 pawpy

would you please attach a test case example with the problem in question ?

edsiper avatar Nov 09 '20 22:11 edsiper

Hello @edsiper

Below is dropped down case of what we are seeing in production.

send log msg to fluent-bit using:

import socket

sock = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
sock.connect(("127.255.0.1", 1020))

bad = "<133>1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad"
bad = bad + '\0' + '\n'

good = "<133>1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% good"
good = good + '\n'

msg = bad + good
sock.send(msg.encode())

Then on fluent-bit i see:

[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] new Unix connection arrived FD=24
[2020/11/11 08:47:20] [trace] [input:syslog:syslog.0 at plugins/in_syslog/syslog_conn.c:73] read()=282 pre_len=0 now_len=282
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 133>1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 33>1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 3>1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: >1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 1 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message:  2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 2020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 020-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: 20-10-29T19:34:48.932381+00:00 i-0c3b7e731bd7c8b6a php.access.log - - [] 2020-10-29 19:34:48 GET 200 468.829 ms 2048 kB 81.05% bad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
...
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: ad
[2020/11/11 08:47:20] [ warn] [input:syslog:syslog.0] error parsing log message with parser 'rfc5424-fluentd'
[2020/11/11 08:47:20] [debug] [input:syslog:syslog.0] unparsed log message: d
[2020/11/11 08:47:20] [trace] [input:syslog:syslog.0 at plugins/in_syslog/syslog_conn.c:83] fd=24 closed connection

Fluent-bit is running with below configurations

root@83322302e579:~/fluent-bit/build# cat /fluent-bit/pawanlocal/fb.conf
[SERVICE]
    Flush               1
    Log_Level           trace

[INPUT]
    Name                syslog
    Buffer_Max_Size 128k
    Tag  syslog-source
    Mode tcp
    Listen 127.255.0.1
    Port 1020
    Parser rfc5424-fluentd

[OUTPUT]
    Name   stdout
    Match  *
root@83322302e579:~/fluent-bit/build# cat /fluent-bit/pawanlocal/parsers.conf
[PARSER]
    Name        rfc5424-fluentd
    Format      regex
    Regex       /\A\<(?<pri>[0-9]{1,3})\>[1-9]\d{0,2} (?<time>[^ ]+) (?<host>[!-~]{1,255}) (?<ident>[!-~]{1,48}) (?<pid>[!-~]{1,128}) (?<msgid>[!-~]{1,32}) (?<extradata>(?:\-|(?:\[.*?(?<!\\)\])+))(?: (?<message>.+))?\z/
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

pawpy avatar Nov 11 '20 08:11 pawpy