New `syslog` sink

[binarylogic]

trafficstars

As per https://github.com/timberio/vector/discussions/6862, we should have a syslog sink that makes it easy to send Syslog formatted logs. This should wrap the socket sink to support the various socket protocols.

[prognant]

Will probably have to support both old (rfc3164) and new (rfc5424).

[wfender]

Hello, Do you have any news about this feature ? It would be a very great and useful feature expected by many people.

[jszwedko]

Hi @wfender . We are working on the groundwork for this in https://github.com/vectordotdev/vector/issues/8617, but don't have a definite ETA yet.

[wfender]

Thank you for this feedback, I look forward to this feature to start experiment Vector in my company. ;)

[syedriko]

Just leaving a comment here that I started working on a new syslog sink, with https://github.com/vectordotdev/vector/pull/7106 as inspiration.

[jszwedko]

Just leaving a comment here that I started working on a new syslog sink, with #7106 as inspiration.

Great! Just a note that we would like to approach this sink a bit differently than #7106 did due to the enhanced codec support that was added to sinks after that PR was opened. Essentially, we'd like to create a syslog codec that can be plugged in with a socket sink (see https://github.com/vectordotdev/vector/tree/master/lib/codecs/src/encoding/format). That will let the syslog sink just be a wrapper around the socket sink with a specific codec configured.

Let me know if that makes sense. Happy to help guide a PR here!

[syedriko]

Just leaving a comment here that I started working on a new syslog sink, with #7106 as inspiration.

Great! Just a note that we would like to approach this sink a bit differently than #7106 did due to the enhanced codec support that was added to sinks after that PR was opened. Essentially, we'd like to create a syslog codec that can be plugged in with a socket sink (see https://github.com/vectordotdev/vector/tree/master/lib/codecs/src/encoding/format). That will let the syslog sink just be a wrapper around the socket sink with a specific codec configured.

Let me know if that makes sense. Happy to help guide a PR here!

Got it, thanks, @jszwedko. Will post a WIP PR as soon as.

[sraka1]

Is this being worked on?

[syedriko]

Is this being worked on?

Yes, it is. Slowly but surely ;)

[haoyu-sgnl]

I would like to hear more about this, need to figure out how to use vector for syslog sink, any additional info would be helpful!!

[git001]

Who is working on this issue, which I can join?

[amarnathpv]

Any updates to where we're with this, please? To support syslog as sink either the code for Syslog via Socket or Syslog as sink itself?

[jszwedko]

Any updates to where we're with this, please? To support syslog as sink either the code for Syslog via Socket or Syslog as sink itself?

No updates yet on our end but we agree this would be a good idea if anyone is motivated to contribute! I still think the best approach would be to implement as a "codec" in Vector. The GELF codec could be a reasonable example to base it on.

[syedriko]

I have a syslog codec implementation, but it is specialized for OpenShift's needs. Let me extract it into a PR against the mainline, it might serve as a starting point for what we want to build.

[amarnathpv]

@syedriko - any luck with the syslog codec?

[syedriko]

@syedriko - any luck with the syslog codec?

@amarnathpv Here you go: https://github.com/vectordotdev/vector/pull/17668

[amarnathpv]

Many thanks @syedriko. That would greatly help!

How can I get this version for testing?

[syedriko]

Many thanks @syedriko. That would greatly help!

How can I get this version for testing?

You can build vector from my repo https://github.com/syedriko/vector.git, off the branch syedriko-syslog-codec. For a start, you can bounce this config

[sources.log_generator]
type = "demo_logs"
format = "json"

[sinks.syslog]
inputs = ["log_generator"]
type = "socket"
address = "0.0.0.0:1514"
mode = "udp"

[sinks.syslog.encoding]
codec = "syslog"
rfc = "rfc5424"
facility = "user"
severity = "debug"
app_name = "myapp"
proc_id = "myproc"
msg_id = "mymsg"

against rsyslogd:

5423.927818395:imudp.c        : imudp.c: recv(4,418),acl:1,msg:<15>1 2023-06-13T15:43:43.927-04:00 - myapp myproc
 mymsg - {"message":"{\"host\":\"251.239.70.77\",\"user-identifier\":\"shaneIxD\",\"datetime\":\"13/Jun/2023:15:43:43
\",\"method\":\"HEAD\",\"request\":\"/wp-admin\",\"protocol\":\"HTTP/1.1\",\"status\":\"302\",\"bytes\":6050,\"referer
\":\"https://names.us/controller/setup\"}","service":"vector","source_type":"demo_logs","timestamp":"2023-06-13T19:43:43.926564381Z"}

Since syslog is a codec just like the other codecs, you can run your socket sink on top of UDP, TCP and TLS.

Syslog message fields can be hardcoded in the sink configuration as above, but they can also be populated from the log event fields. Here's an example:

[sources.log_generator]
type = "demo_logs"
format = "json

[transforms.parse_demo_logs]
inputs = ["log_generator"]
type = "remap"
source = '''
. = parse_json!(string!(.message))
'''

[sinks.foobar]
inputs = ["parse_demo_logs"]
type = "socket"
address = "0.0.0.0:1514"
mode = "udp"

[sinks.foobar.encoding]
codec = "syslog"
rfc = "rfc5424"
facility = "user"
severity = "debug"
app_name = "myapp"
proc_id = "myproc"
msg_id = "$$.message.host"

, which produces

1343.221646458:imudp.c        : imudp.c: recv(4,300),acl:1,msg:<15>1 2023-06-13T20:09:03.221-04:00 - myapp myproc 137.212.170.113 - {"bytes":9699,"datetime":"13/Jun
/2023:20:09:03","host":"137.212.170.113","method":"PATCH","protocol":"HTTP/2.0","referer":"https://for.us/controller
/setup","request":"/controller/setup","status":"301","user-identifier":"ahmadajmi"}

The "$$.message.host" syntax is there for legacy reasons. "host" is the name of the log event field to populate the RFC 5424 msg_id field with.

[amarnathpv]

Awesome, thanks Sergej @sydreko

[pznamensky]

hey @syedriko, since https://github.com/vectordotdev/vector/pull/17668 seems to be working fine, how do you feel about converting it from draft in to PR?

[syedriko]

@pznamensky I'm glad to hear ^^^, it's working pretty well in OpenShift, too. I'll get to it, too many balls to juggle.

[gaby]

@syedriko @jszwedko Any update on this? I was going to use Vector to emit syslog from multiple sources and ran into a road-block because of the lack of a Syslog sink.

Basically I sent syslof from multiple sources into vector and parse these as JSON, but I also need to forward the original Syslog to other follow-on processes.

[jszwedko]

@syedriko @jszwedko Any update on this? I was going to use Vector to emit syslog from multiple sources and ran into a road-block because of the lack of a Syslog sink.

Basically I sent syslof from multiple sources into vector and parse these as JSON, but I also need to forward the original Syslog to other follow-on processes.

Nothing yet unfortunately. If anyone wants to take a stab at contributing this, I think the path would look like:

• Create a syslog encoder
• Create a syslog sink that is just the socket sink with the syslog encoder hardcoded

[gaby]

@syedriko @jszwedko Any update on this? I was going to use Vector to emit syslog from multiple sources and ran into a road-block because of the lack of a Syslog sink. Basically I sent syslof from multiple sources into vector and parse these as JSON, but I also need to forward the original Syslog to other follow-on processes.

Nothing yet unfortunately. If anyone wants to take a stab at contributing this, I think the path would look like:

• Create a syslog encoder
• Create a syslog sink that is just the socket sink with the syslog encoder hardcoded

What about #17668 ? It seems to cover those.

[jszwedko]

@syedriko @jszwedko Any update on this? I was going to use Vector to emit syslog from multiple sources and ran into a road-block because of the lack of a Syslog sink. Basically I sent syslof from multiple sources into vector and parse these as JSON, but I also need to forward the original Syslog to other follow-on processes.

Nothing yet unfortunately. If anyone wants to take a stab at contributing this, I think the path would look like:

• Create a syslog encoder
• Create a syslog sink that is just the socket sink with the syslog encoder hardcoded

What about #17668 ? It seems to cover those.

That looks like a good start! The original author seems to have fallen off unfortunately, but it should serve as a good base for a new PR.

[syedriko]

@syedriko @jszwedko Any update on this? I was going to use Vector to emit syslog from multiple sources and ran into a road-block because of the lack of a Syslog sink. Basically I sent syslof from multiple sources into vector and parse these as JSON, but I also need to forward the original Syslog to other follow-on processes.

Nothing yet unfortunately. If anyone wants to take a stab at contributing this, I think the path would look like:

• Create a syslog encoder
• Create a syslog sink that is just the socket sink with the syslog encoder hardcoded

What about #17668 ? It seems to cover those.

That looks like a good start! The original author seems to have fallen off unfortunately, but it should serve as a good base for a new PR.

@jszwedko I haven't gone anywhere, just been swamped. It would be great to get https://github.com/vectordotdev/vector/pull/17668 reviewed so I can start addressing the feedback.

[gaby]

@syedriko Awesome to hear, can you update you branch and fix merge conflicts. Also mark it as ready, right now it shows as "Draft"

[syedriko]

@syedriko Awesome to hear, can you update you branch and fix merge conflicts. Also mark it as ready, right now it shows as "Draft"

Done

[gaby]

@syedriko Awesome to hear, can you update you branch and fix merge conflicts. Also mark it as ready, right now it shows as "Draft"

Done

Thank you!

@jszwedko This should be step forward for getting thus feature :-)

[jszwedko]

Thanks @syedriko ! We'll take a look and leave a review.