Syslog output doesn't send data to Qradar

[hoooli]

Bug Report

Describe the bug I'm trying to set up sending logs to QRadar via syslog output, but only TCP info logs (about connection) are arriving in QRadar (without the logs I'm sending). My Fluentbit is running as a sidecar and in the file which I read with tail plugin and all other outputs, there are regular messages, but they're missing in QRadar.

My fluentbit version: 2.2.1

My log format is plain text like: SECURITY timestamp message

My confguration

[SERVICE]
    Log_Level     debug
[INPUT]
    Name              tail
    Tag               app.${cluster}.${namespace}.<filename>
    Tag_Regex         /app/log/(?<filename>.+).log$
    Path              /app/log/*.log
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Buffer_Max_Size   5MB
    Refresh_Interval  60
[OUTPUT]
    Name         forward
    Match        *
    Host         ${fluentd-host}
    Port         ${fluentd-port}
    Retry_Limit  5
[OUTPUT]
    Name        syslog
    Match       *
    Host        ${qradar_ip}
    Port        514
    Mode        tcp
[OUTPUT]
    Name        loki
    Match       app.*
    Host        ${loki_url}
    Port        ${loki_port}
    Retry_Limit  1

At the start there is info about plugins:

[2024/05/02 12:51:17] [ info] Configuration:
[2024/05/02 12:51:17] [ info]  flush time     | 1.000000 seconds
[2024/05/02 12:51:17] [ info]  grace          | 5 seconds
[2024/05/02 12:51:17] [ info]  daemon         | 0
[2024/05/02 12:51:17] [ info] ___________
[2024/05/02 12:51:17] [ info]  inputs:
[2024/05/02 12:51:17] [ info]      tail
[2024/05/02 12:51:17] [ info] ___________
[2024/05/02 12:51:17] [ info]  filters:
[2024/05/02 12:51:17] [ info] ___________
[2024/05/02 12:51:17] [ info]  outputs:
[2024/05/02 12:51:17] [ info]      forward.0
[2024/05/02 12:51:17] [ info]      syslog.1
[2024/05/02 12:51:17] [ info]      loki.2
[2024/05/02 12:51:17] [ info] ___________

and correct connection:

[2024/05/02 12:51:17] [debug] [forward:forward.0] created event channels: read=29 write=30
[2024/05/02 12:51:17] [debug] [syslog:syslog.1] created event channels: read=41 write=42
[2024/05/02 12:51:17] [ info] [output:forward:forward.0] worker #0 started
[2024/05/02 12:51:17] [ info] [output:syslog:syslog.1] setup done for qradar_ip:514 (TLS=off)
[2024/05/02 12:51:17] [debug] [loki:loki.2] created event channels: read=47 write=48
[2024/05/02 12:51:17] [ info] [output:forward:forward.0] worker #1 started
[2024/05/02 12:51:17] [ info] [output:loki:loki.2] configured, hostname=loki_url:loki_port
[2024/05/02 12:51:17] [debug] [router] match rule tail.0:forward.0
[2024/05/02 12:51:17] [debug] [router] match rule tail.0:syslog.1
[2024/05/02 12:51:17] [debug] [router] match rule tail.0:loki.2
[2024/05/02 12:51:17] [ info] [sp] stream processor started

And when new log is found, I see other outputs in fluentbit log

[2024/05/02 13:41:38] [debug] [input:tail:tail.0] inode=131968, /app/log/app.log, events: IN_MODIFY 
[2024/05/02 13:41:38] [debug] [input chunk] update output instances with new chunk size diff=2200, records=1, input=tail.0
[2024/05/02 13:41:39] [debug] [task] created task=0x7fc1c5e36f00 id=0 OK
[2024/05/02 13:41:39] [debug] [output:forward:forward.0] request 2200 bytes to flush
[2024/05/02 13:41:39] [debug] [output:forward:forward.0] task_id=0 assigned to thread #1
[2024/05/02 13:41:39] [debug] [upstream] KA connection #66 to fluentd_ip:fluentd_port is connected
[2024/05/02 13:41:39] [debug] [upstream] KA connection #66 to fluentd_ip:fluentd_port is now available
[2024/05/02 13:41:39] [debug] [out flush] cb_destroy coro_id=0
[2024/05/02 13:41:39] [debug] [upstream] KA connection #67 to loki_url:loki_port is connected
[2024/05/02 13:41:39] [debug] [http_client] not using http_proxy for header
[2024/05/02 13:41:39] [debug] [upstream] KA connection #65 to qradar_ip:514 is connected
[2024/05/02 13:41:39] [debug] [upstream] KA connection #65 to qradar_ip:514 is now available
[2024/05/02 13:41:39] [debug] [out flush] cb_destroy coro_id=1
[2024/05/02 13:41:39] [debug] [output:loki:loki.2] loki_url:loki_port, HTTP status=204
[2024/05/02 13:41:39] [debug] [upstream] KA connection #67 to loki_url:loki_port is now available
[2024/05/02 13:41:39] [debug] [out flush] cb_destroy coro_id=1
[2024/05/02 13:41:39] [debug] [task] destroy task=0x7fc1c5e36f00 (task_id=0)
[2024/05/02 13:42:09] [debug] [upstream] drop keepalive connection #-1 to fluentd_ip:fluentd_port (keepalive idle timeout)
[2024/05/02 13:42:09] [debug] [upstream] drop keepalive connection #-1 to qradar_ip:514 (keepalive idle timeout)
[2024/05/02 13:42:09] [debug] [upstream] drop keepalive connection #-1 to loki_ip:loki_port (keepalive idle timeout)
[2024/05/02 13:42:17] [debug] [input:tail:tail.0] scanning path /app/log/*.log

But there is no syslog output...only 2 lines about connection to qradar_ip but no flush info or something.

I tried to add syslog_message_key (value log, but i tried message also) but without luck.
Changing format didnt help. Is there anything that I need to add to send the data?

Thank you

[patrick-stephens]

Can you reproduce with the latest 3.0 versions?

[hoooli]

@patrick-stephens Yes, I tried the same scenario on version 3.0.2 - the situation was exactly the same, no information about syslog output in log and no data in Qradar. Do you have any suggestions on what to try next?

[patrick-stephens]

tcpdump or similar to see what's going on with the packets - also please follow the issue template as you've not indicated your platform and other useful info. If it's in a container then maybe the packets are not being routed outside.

[agup006]

I’d try the tcp or udp output instead first

[hoooli]

It looks like there was an issue on the qradar side, I added two fields syslog_message_key (required) and syslog_hostname_key and it works like a charm. In the logs, even with a functional setup, there is no output:syslog section which is a bit confusing, but it works so it's not a problem :) Thank you guys.