Improvement/plugging crowdsec in an existing log flow with fluent protocol

[athoune]

Lots of logs reader

It's easy to use Crowdsec with a classical UNIX web server (Apache or Nginx), tailing a log file is universal. Universal, but logstash/filebeat/fluent-bit/vector already does that, and parse logs with regexp.

Using standard log flow

How can I plug Crowdsec in already existing log flow? What about fluent protocol, blessed by the CNCF?

Fluent input for crowdsec

With a fluent input, crowdsec can be plugged in an existing workflow, using already parsed log (never regexp twice) and can push its flow to another fluent service.

[buixor]

@athoune that's a great suggestion, especially as we're revamping the acquisition to support more data sources !

[athoune]

Opentelemetry try to normalize lots of things, but not the log flow, outside the application.

I don't trust in syslog (udp, small message, cheap routing with facility, unauthenticated…) but it's a standard. Logstash protocol is nice, but not documented, with only reverse engineered hack for server side implementation.

Fluentd is boring, but fluent-bit is useful, lots of service can speak the fluent protocol.

The officiel go implementation doesn't include the server part, just some tool for testing the client part.

It's easy to implement the server part, most of the job is done by msgpack : https://github.com/factorysh/fluent-server

With a nice acquisition interface, it will be easy to implement more inputs, I'll watch the PR.

[chibenwa]

Hello.

:+1: for this: that is something I would need...

If not implemented yet, I could potentially get this contributed.

[rr404]

Hello @chibenwa, we'd appreciate your contribution very much, I'll get in touch with you by email. Talk to you soon

We'll keep the issue update following that call.