fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

Please add TLS/SSL support to syslog input plugin.

Open davidnewhall opened this issue 4 years ago • 26 comments

Is your feature request related to a problem? Please describe. I'm only able to send syslog messages to fluent-bit across a network using plain text UDP or TCP.

Describe the solution you'd like Please add TLS support to the syslog input plugin.

Describe alternatives you've considered I'm currently running a sidecar container with nginx to terminate SSL and write to a syslog socket file (into fluent-bit). I'd like to not have to run nginx, and instead pass an SSL cert/CA/key into fluent-bit directly.

Additional context Amazon requires all service-to-service communications to be encrypted in flight. Per company policy, this missing feature means Amazon employees have a difficult time utilizing the syslog input plugin [without an nginx sidecar].

davidnewhall avatar Aug 31 '20 04:08 davidnewhall

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Mar 24 '21 02:03 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity.

github-actions[bot] avatar Mar 30 '21 02:03 github-actions[bot]

rip.

davidnewhall avatar Mar 30 '21 02:03 davidnewhall

Can we please get this issue re-opened? It is a legitimate feature request.

smlx avatar Sep 24 '21 02:09 smlx

+1 high priority

edsiper avatar Sep 24 '21 04:09 edsiper

https://docs.fluentbit.io/manual/pipeline/outputs/syslog

...It supports RFC3164 and RFC5424 formats through different transports such as UDP, TCP or TLS.

There is already the parameter mode that supports tls.

markuman avatar Oct 17 '22 06:10 markuman

@markuman How do you provide certificates and key files path with that? couldn't see any parameter for that.

prashansag62 avatar Nov 29 '22 10:11 prashansag62

I'm not sure if that tls mode option ever worked, I don't remember looking at it before I changed the code but right now in 2.x you can put this in the input plugin section :

    tls             on
    tls.vhost  vhost_name
    tls.crt_file path_to_cert
    tls.key_file path_to_key

You can combine TLS with both tcp and uds modes (except uds has to be in stream mode) which means setting mode to tcp or unix_tcp (yes, the name for stream mode unix sockets isn't great).

You can find more information about the current state of TLS support in this page https://docs.fluentbit.io/manual/administration/security

leonardo-albertovich avatar Nov 29 '22 15:11 leonardo-albertovich

@leonardo-albertovich , will this work in any input plugin ? I am getting following error when using this code :

[INPUT]
Name         syslog
    Parser       syslog-rfc3164
    Listen       0.0.0.0
    Port         3020
    Mode      tcp
    Tag          syslog
    tls          on
    tls.verify   off
    tls.crt_file /fluent-bit/etc/ssl/certs/ssl-cert.pem
    tls.key_file /fluent-bit/etc/ssl/certs/ssl-key.pem

image image

prashansag62 avatar Dec 06 '22 18:12 prashansag62

Yes it does, how is your syslog client configured? I used that setup a lot for testing (all combinations of transport and encryption / plaintext actually).

leonardo-albertovich avatar Dec 07 '22 00:12 leonardo-albertovich

Sure @leonardo-albertovich.Following is the conf file of syslof client.

# Load imfile
module(load="imfile")

# Tag accounting logs as local2:accounting
input(type="imfile"
    File="/var/spool/accounting/*"
    Tag="accounting"
    Severity="info"
    Facility="local2"
)

# Tag scheduler logs as local2:scheduler
input(type="imfile"
    File="/var/spool/sched_logs/*"
    Tag="scheduler"
    Severity="info"
    Facility="local2"
)

global(DefaultNetstreamDriverCAFile="/etc/ssl/certs/DigiCert.pem")

# Forward Syslogs to fluentbit load balancer
*.*;local2.none action(type="omfwd" target="10.312.0.60" port="3020" protocol="tcp" action.resumeRetryCount="100" 
queue.type="linkedList" queue.size="10000" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon")

# Forward Accounting & Scheduler logs to fluentbit load balancer.
local2.* action(type="omfwd" target="10.312.0.60" port="3021" protocol="tcp" action.resumeRetryCount="100" 
queue.type="linkedList" queue.size="10000" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon")

Also I didnt specify vhost in fluentbit input plugin. Is it must for making tls work on syslog? If yes, I am not sure what should be exactly provided here. tls.vhost vhost_name

prashansag62 avatar Dec 07 '22 15:12 prashansag62

I just replicated your setup locally (ubuntu 22.04) and it worked as expected so I'll paste my configurations in here so we can compare both of them and figure out what's wrong :

rsyslog.conf (this is a verbatim copy of yours, I just replaced the host and port)

*.=info;*.=notice;*.=warn;\
       auth,authpriv.none;\
       cron,daemon.none;\
       mail,news.none          action(type="omfwd" target="127.0.0.1" port="9999" protocol="tcp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon")

fluent-bit.conf

[SERVICE]
    flush 1
    grace 1
    log_level debug
    parsers_file parsers.conf

[INPUT]
    name syslog
    listen 0.0.0.0
    port 9999
    mode tcp
    parser syslog-rfc3164
    tls.verify no
    tls on
    tls.crt_file self_signed.crt 
    tls.key_file self_signed.key

[OUTPUT]
    name stdout
    match *

I verified that the tls.vhost can be omitted with no issues, what I had to adapt though was the parser because the default one didn't work in my setup (but fluent-bit explicitly says that)

Additionally, before running fluent-bit I tested rsyslog with a both a plaintext and tls listener running these commands respectively:

nc -k -l 9999 | hexdump -C

openssl s_server -cert self_signed.crt -key self_signed.key -port 9999 | hexdump -C

Maybe that's a test you can make in order to determine if your syslog daemon is able to reach the endpoint as expected.

Btw, I noticed the background color of your terminal is blue, are you by chance running fluent-bit in windows? If so, could you tell me which version so I can replicate that part of the setup as well?

leonardo-albertovich avatar Dec 08 '22 03:12 leonardo-albertovich

@leonardo-albertovich , my fluentbit instance is runnong on azure kubernetes cluster, and blue color is because of azure command line prompt. Because of the above showed errors, fluentbit containers are in crashloopback state. And since pods aren't getting in running state, I think syslog daemon is not able to connect with endpoint.

when I am running systemctl status rsyslog , I am getting following errors action 'action-3-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one >Dec 06 omfwd: remote server at 10.312.0.60:3021 seems to have closed connection. This often happens when the remote peer (or >

Also can you tell me which version of fluentbit are you using? I am using 2.0.5

prashansag62 avatar Dec 08 '22 04:12 prashansag62

I'm using master but it should be almost the same as 2.0.6

leonardo-albertovich avatar Dec 08 '22 05:12 leonardo-albertovich

hm I hit a similar problem using fluent-bit 2.0.6

[2022/12/16 09:59:31] [error] [io_tls] handshake connection #40 to tcp://10.127.19.219:21626 timed out after 10 seconds
[2022/12/16 09:59:31] [debug] [downstream] connection #40 failed
[2022/12/16 09:59:31] [error] [input:syslog:syslog.0] could not accept new connection

config

[SERVICE]
    flush 1
    grace 1
    log_level debug
    parsers_file /etc/fluent-bit/parsers.conf

[INPUT]
    name syslog
    listen 0.0.0.0
    port 9997
    mode tcp
    parser syslog-rfc3164
    tls.verify no
    tls on
    tls.crt_file /etc/ssl/influxdb.aws.xxx.de.crt
    tls.key_file /etc/ssl/influxdb.aws.xxx.de.key

[OUTPUT]
    name stdout
    match **

For the cert I've did

openssl req -nodes -new -x509 -keyout influxdb.aws.xxx.de.key -out influxdb.aws.xxx.de.crt -days 3650

and imported the same key and crt file also in our client (opnsense)

opnsense syslog tls

syslog error message in opnsense is just

Syslog connection failed; fd='27', server='AF_INET(10.112.60.131:9997)', error='Connection refused (61)', time_reopen='60'

any ideas or hints @leonardo-albertovich?

markuman avatar Dec 16 '22 10:12 markuman

This is the configuration that opnsense creates

### log target 98c20079-7c9e-4ce5-8081-3da30e814a64 :  ####
filter f_98c200797c9e4ce580813da30e814a64_program {
    program("audit") or program("firewall") or program("openvpn")
};
filter f_98c200797c9e4ce580813da30e814a64_level {
    level(info,notice,warn,err,crit,alert,emerg)
};

destination d_98c200797c9e4ce580813da30e814a64 {
    network(
        "influxdb.aws.xxx.de"
        transport("tls")
        port(9997)
        ip-protocol(4)
        persist-name("98c200797c9e4ce580813da30e814a64")
        tls(
                ca-file("/etc/ssl/cert.pem")
                key-file("/usr/local/etc/syslog-ng/cert.d/98c200797c9e4ce580813da30e814a64.key")
                cert-file("/usr/local/etc/syslog-ng/cert.d/98c200797c9e4ce580813da30e814a64.crt")
        )
    );
};

log {
    source(s_all);
    filter(f_98c200797c9e4ce580813da30e814a64_program);
    filter(f_98c200797c9e4ce580813da30e814a64_level);
    destination(d_98c200797c9e4ce580813da30e814a64);
};

markuman avatar Dec 16 '22 10:12 markuman

I get it work with a custom config now

destination influx {
    network(
        "influxdb.xxx.de"
        transport("tls")
        port(9997)
        ip-protocol(4)
        tls(
                ca-file("/etc/ssl/cert.pem")
                ca_dir("/etc/ssl/certs/*")
                peer_verify(optional-trusted)
        )
    );
};

log {
    source(s_all);
    destination(influx);
};

markuman avatar Dec 16 '22 11:12 markuman

Great, I was wondering if verification was part of the issue. I was writing a message asking for more details when this popped up.

What's clear to me in this case is neither system is doing a great job communicating the issue to the operator which can be pretty frustrating.

A PR to add that to fluent-bit would be great, does anyone think they would be able to do so with some guidance?

leonardo-albertovich avatar Dec 16 '22 11:12 leonardo-albertovich

Another thing I've observed....

My debug/dev config works fine.

[SERVICE]
    flush 1
    grace 1
    log_level debug
    parsers_file /etc/fluent-bit/parsers.conf
    Streams_File /etc/fluent-bit/stream_processor.conf

[INPUT]
    name syslog
    listen 0.0.0.0
    port 9997
    mode tcp
    parser syslog-rfc3164
    tls.verify no
    tls on
    tls.crt_file /etc/ssl/xxx.de.crt
    tls.key_file /etc/ssl/xxx.de.key

[OUTPUT]
    name stdout
    match vpn.syslog

[OUTPUT]
    Name cloudwatch_logs
    Match vpn.syslog
    region eu-central-1
    log_group_name /var/log/opnsense
    log_stream_name vpn
    auto_create_group On
    log_retention_days 14

but in production, it fails. it is just not processed and forwarded.
the config is build with ansible. So I guess copy/paste isn't the issue here

grep -ic input /etc/fluent-bit/config 
10

the main difference is, that there are more inputs defined and that there are two stream processes defined and also in use,

[STREAM_TASK]
    Name    cert.sp
    Exec    CREATE STREAM cert WITH (tag='cert.log') AS SELECT MESSAGE as log from TAG:'cert.raw';

[STREAM_TASK]
    Name    vpn.sp
    Exec    CREATE STREAM vpn WITH (tag='vpn.syslog') AS SELECT * from TAG:'syslog.0' where host = 'vpn.aws.xxx.de';

while the debug/dev setup is also loaded both, but only one is used.
any idea about that @leonardo-albertovich?

markuman avatar Dec 16 '22 12:12 markuman

That seems to be related to the stream processor which is something I have absolutely no experience with. Does the same thing happen if you use two stdout instances as outputs? Do you think you would be able to minimize the reproduction case?

leonardo-albertovich avatar Dec 16 '22 16:12 leonardo-albertovich

It works now. It was a network permission issue. But now fluent-bit is dying from time to time

[0] vpn.syslog: [1671554537.000000000, {"pri"=>"134", "time"=>"Dec 20 16:42:17", "host"=>"vpn.aws.xxx.de", "ident"=>"filterlog", "pid"=>"86931", "message"=>"57,,,c1eff64cbafdd6b80448f92cd4aff7e5,enc0,match,pass,out,4,0x0,,63,1978,0,none,17,udp,64,10.103.0.6,10.112.30.170,54928,53,44"}]
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [msgpack2json] unknown msgpack type 69
[2022/12/20 15:42:18] [ warn] [msgpack2json] unknown msgpack type 69
[2022/12/20 15:42:18] [ warn] [output:cloudwatch_logs:cloudwatch_logs.1] Discarding massive log record
[2022/12/20 15:42:18] [ warn] [time] unknown ext type. type=21 size=8
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [msgpack2json] unknown msgpack type 69
[2022/12/20 15:42:18] [ warn] [msgpack2json] unknown msgpack type 69
[2022/12/20 15:42:18] [ warn] [output:cloudwatch_logs:cloudwatch_logs.1] Discarding massive log record
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [ warn] [pack] invalid UTF-8 bytes found, skipping bytes
[2022/12/20 15:42:18] [engine] caught signal (SIGSEGV)
#0  0xb0e1e54e011b      in  msgpack_unpack_next() at ???:0
#1  0xb0e1e5358deb      in  process_and_send() at ???:0
#2  0xb0e1e5355b47      in  ???() at ???:0
#3  0xb0e1e52d5ebb      in  ???() at ???:0
#4  0xffffffffffffffff  in  ???() at ???:0
Aborted

markuman avatar Dec 20 '22 15:12 markuman

Increasing the debug level doesn't bring more information @leonardo-albertovich

[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=277
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=270
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=277
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=278
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=222
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=277
[2022/12/20 15:46:57] [engine] caught signal (SIGSEGV)
#0  0xb0e0e6c24383      in  ???() at ???:0
#1  0xf8c03c1ed5c7      in  ???() at ???:0
#2  0xf8c03c255d1b      in  ???() at ???:0
#3  0xffffffffffffffff  in  ???() at ???:0
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=278
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=269
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=226
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=227
[2022/12/20 15:46:57] [debug] [output:cloudwatch_logs:cloudwatch_logs.3] task_id=-1945943690 assigned to thread #0
[2022/12/20 15:46:57] [debug] [task] destroy task=0xf8c034178640 (task_id=1)
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=218
[2022/12/20 15:46:57] [debug] [task] destroy task=0xf8c034178b60 (task_id=0)

I've added some buffer params. But that also changes nothing

[INPUT]
    name syslog
    listen 0.0.0.0
    port 9997
    mode tcp
    parser syslog-rfc3164
    Buffer_Chunk_Size   32000
    Buffer_Max_Size     64000
    Receive_Buffer_Size 512000
    tls.verify no
    tls on
    tls.crt_file /etc/ssl/xxx.de.crt
    tls.key_file /etc/ssl/xxx.de.key

markuman avatar Dec 20 '22 15:12 markuman

First of all I'm glad you got it to "work" but it seems like we still have some stuff to figure out.

The amount of invalid UTF-8 bytes found entries doesn't look good at all. Do you think you would be able to capture those messages so we can try to reproduce it in a development environment? Does the same thing happen if you disable TLS? (I understand you might not be able to do so due to your security policy but I have to ask at least)

I'm trying to come up with a way to get a better picture of what's going on in your deployment, feel free to ping me in slack if you prefer to or let me know if you have any ideas.

leonardo-albertovich avatar Dec 21 '22 11:12 leonardo-albertovich

Just to reaffirm that the above latest config does work. You can easily test the syslog using simple bash script.

You can create a test log with Ubuntu/Debian using gnutls-cli

Install

sudo apt-get install gnutls-bin

sendTLSLog.sh

ts=$(date "+%b %d %T")
echo "<142>$ts $HOSTNAME test: this is a test on test $RANDOM." | \
    gnutls-cli log.example.com --port=6514 --x509cafile=./root.ca.crt

Send log

bash sendTLSLog.sh

Here is a config snippet example below.

    [OUTPUT]
        Name      stdout
        Match     syslog.6514.tcp

    [INPUT]
        Name     syslog
        Parser   syslog-rfc3164
        Tag    syslog.6514.tcp
        Listen   0.0.0.0
        Port     6514
        Mode     tcp
        tls.verify no
        tls on
        tls.crt_file /app/certs/log.example.com.crt
        tls.key_file /app/certs/log.example.com.key

You may see some errors like [2023/06/24 22:25:44] [error] [tls] error: unexpected EOF But those are commonly tcp health probes from loadbalancer, liveness probes or other health checks. You can look at modifying or disabling these probes if you want.

jeremybusk avatar Jun 24 '23 22:06 jeremybusk

@leonardo-albertovich I have one doubt here why do tls.verify no every time, what is diff between tls.verify no and tls.verify on

SharathSunny avatar Feb 15 '24 10:02 SharathSunny

Increasing the debug level doesn't bring more information @leonardo-albertovich

[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=277
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=270
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=277
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=278
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=222
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=277
[2022/12/20 15:46:57] [engine] caught signal (SIGSEGV)
#0  0xb0e0e6c24383      in  ???() at ???:0
#1  0xf8c03c1ed5c7      in  ???() at ???:0
#2  0xf8c03c255d1b      in  ???() at ???:0
#3  0xffffffffffffffff  in  ???() at ???:0
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=278
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=269
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=226
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=227
[2022/12/20 15:46:57] [debug] [output:cloudwatch_logs:cloudwatch_logs.3] task_id=-1945943690 assigned to thread #0
[2022/12/20 15:46:57] [debug] [task] destroy task=0xf8c034178640 (task_id=1)
[2022/12/20 15:46:57] [debug] [input chunk] update output instances with new chunk size diff=218
[2022/12/20 15:46:57] [debug] [task] destroy task=0xf8c034178b60 (task_id=0)

I've added some buffer params. But that also changes nothing

[INPUT]
    name syslog
    listen 0.0.0.0
    port 9997
    mode tcp
    parser syslog-rfc3164
    Buffer_Chunk_Size   32000
    Buffer_Max_Size     64000
    Receive_Buffer_Size 512000
    tls.verify no
    tls on
    tls.crt_file /etc/ssl/xxx.de.crt
    tls.key_file /etc/ssl/xxx.de.key

@leonardo-albertovich if possible can you provide the respective output configuration from where the log's are forwarded to this input plugin

SharathSunny avatar Feb 16 '24 03:02 SharathSunny