udp json input support

[nvx]

Is your feature request related to a problem? Please describe. There is currently a json tcp input, but no corresponding option for udp

Describe the solution you'd like Having the ability to consume json events sent over udp would be beneficial

[edsiper]

would you please elaborate your use case ?

[nvx]

Reliably transporting logs sent from applications via UDP.

In some instances it is desirable for the application to emit logs via UDP rather than handle reconnection/buffering/blocking/etc in the application where dropping logs in the event of logging infrastructure failures are acceptable. The main problem with this approach is that sending UDP over the network offers very little guarantees.

A pattern I have used is ingesting UDP logs over localhost to a local agent, which in turn then takes care of buffering and reliably sending over the network. This pattern can be achieved today with the in_udp fluentd input, Logstash, or nxlog. The downside is the former two agents are quite heavyweight, while nxlog meets the lightweight requirement it's capabilities are rather limited compared to eg fluent-bit.

The use-case is similar to syslog ingestion over tcp vs udp which I note fluent-bit supports both today.

[newrushbolt]

I got pretty much the same case that @nvx, so i'd like to vote up for this feature.

[proligde]

Same scenario here - are there any plans to make that happen?

[dontspamterry]

Likewise, would be nice to have UDP input

[p53]

+1 for UDP input, we would like to replace rsyslog forwarder with fluent-bit

[nvx]

Is there still any information needed? I notice the awaiting for user label is still on this @edsiper

[jwitrick]

Is there any timeline/update for this? UDP input would be a HUGE win for streaming logs with fluent bit.

[bluebike]

Actually in_syslog could be used as generic UDP input.. .just by giving suitable parser. I thought that if we just take in_syslog and copy it to as in_udp (some search/replace). Then add right method to parse tag name from message... and we have generic UDP input.

Also this could be used for something like in_influxdb for influxdb line protocol with UDP. In that case it would better to InfluxDB line protocol parser (regex is not really enough for that).

[JimHagan]

We have this use case. I'm an engineer at New Relic and we would like to be able to offer this option to our users who use fluentbit as a forwarder.

[bluebike]

@JimHagan Few weeks ago I copied in_syslog to in_udp ... replaces syslog to udp... and did some testing (didn't have to time do more...) I could try to make PR in few days..

[bluebike]

I made PR #2414 in_udp module... @nvx @JimHagan You could have test... or give comments if module would cover your needs.

I have tested it only in my macOS (10.13) so far.

[frzb]

One further use case:

Sending kernel and early user space (initramfs) logs via the netconsole kernel module over UDP before any other logging daemon is running during system boot.

I use that with Coinboot for a huge number of diskless nodes without any KVM over IP capabilities for debugging early boot stages. These kernel message don't align with RFC 3164 or RFC 5424 and so fail to be ingested by the in_syslog module.

[parera10]

@bluebike What's about your PR? Is it staled? I would appreciate to have this feature on fluent-bit.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[nvx]

ping

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[nvx]

ping

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[nvx]

still relevant

[mokaton]

ping

[achetronic]

Hello mates, another usecase here is to replace logstash when it is configured with the udp input. Is the PR ready to merge?

[leonardo-albertovich]

I don't know about the PR but a while ago while working in the downstream provider I created both a UDS and UDP input plugins.

These were created for testing and are basically fleshed out copies of the TCP input plugin so they could need a bit of tuning but don't hesitate to ping me so I can fix any issues that arise.

Note : There is no official documentation for in_udp and in_unix_socket because as I said those were created by me for testing purposes mostly but I'm sure they will get properly documented in due time if they prove to be useful.

[achetronic]

@leonardo-albertovich Hello! :) thank you for the response. I saw this src after you told me it exists but I would appreciate if you could help with the documentation about how to configure the input, etc, to test it myself :)

[leonardo-albertovich]

This should get you going :

[INPUT]
    name udp
    listen 127.0.0.1
    port 9990

[INPUT]
    name udp
    listen 127.0.0.1
    port 9991
    format none

[INPUT]
    name udp
    listen 127.0.0.1
    port 9992
    format none
    separator |

With this config fluent-bit will be waiting for :

1. json formatted data on port 9990
2. raw data separated by new lines on port 9991
3. raw data separated by pipes on port 9992

Just keep in mind that if you send multiple entries in a single packet you need to send one separator per entry (ie. if you send 1|2 to port 9992 fluent-bit will only ingest 1 but if you send 1|2| it will ingest both entries as individual records which I'm not really sure is the intended operation mode but if it's not then we should check in_tcp to determine if it behaves in the same way and then report the issue (that way I'll have to fix in_tcp and will fix both in_udp and in_unix_socket as a side effect).

Hope it helps!

[bluebike]

@leonardo-albertovich good I can remove my staled PR #2414

[achetronic]

This should get you going :

[INPUT]
    name udp
    listen 127.0.0.1
    port 9990

[INPUT]
    name udp
    listen 127.0.0.1
    port 9991
    format none

[INPUT]
    name udp
    listen 127.0.0.1
    port 9992
    format none
    separator |

With this config fluent-bit will be waiting for :

1. `json` formatted data on port 9990

2. raw data separated by new lines on port 9991

3. raw data separated by pipes on port 9992

Just keep in mind that if you send multiple entries in a single packet you need to send one separator per entry (ie. if you send 1|2 to port 9992 fluent-bit will only ingest 1 but if you send 1|2| it will ingest both entries as individual records which I'm not really sure is the intended operation mode but if it's not then we should check in_tcp to determine if it behaves in the same way and then report the issue (that way I'll have to fix in_tcp and will fix both in_udp and in_unix_socket as a side effect).

Hope it helps!

Oh, thank you! I will test it and give some feedback. If everything is ok, I can collaborate in the documentation if needed

[leonardo-albertovich]

I'm sure it would be highly appreciated.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[nvx]

It sounds like this can be closed other than perhaps some docs describing how to use it?