liblognorm
liblognorm copied to clipboard
rsyslog
Compatibility with grok rules
I was wondering if it's technically possible to have a converter that would translate grok rules and dictionaries to their liblognorm equivalent. At least for the known stuff like IPs, word=NOTSPACE, etc. Or better still, have liblognorm read grok rules and interpret them as native stuff.
I think there's some manual work needed to make equivalent types to those grok rules that are built on raw regular expressions, but for the rest it should be doable. Though the end result might be too messy and unmaintainable.
So maybe it just makes sense to go through all the rules and dictionaries, pull out the regexes and then just have liblognorm use those regular expressions. Sure, it will be a lot slower than using native rules, but it would allow someone to re-use stuff already on the Internet. And maybe convert to native rules later on if performance becomes an issue.
https://github.com/SinaMSRE/rsyslog-mmgrok
I wrote a rsyslog plugin to use logstash grok pattern.
This is very interesting. Do you mind to merge this to contrib? If so, a PR would be good - but I could also pull from your repo.
good to have as an option, how does it perform? (especially with large numbers of rules)
On Wed, 25 Nov 2015, 饶琛琳 wrote:
https://github.com/SinaMSRE/rsyslog-mmgrok
I wrote a rsyslog plugin to use logstash grok pattern.
There isn't a exists libgrok rpm in EPEL, need to rpmbuild by myself, so I didn't send PR to rsyslog directly. But I can send PR anytime if you like.
Haven't do a performance benchmask, maybe later.
The existence of a RPM is not a necessary precondition, we have many modules which require something that's not packaged, especially in the contrib tree. So fell free to do a PR. Once I have some more time (I guess around March), I'll probably check if we can make this an officially supported module. I think freedom of choice would be good here.
@davidelang performance will be bad by our standards, because that is how it is with the grok engine. I still think there are use cases in lower traffic environments.