Log support like Logtash

[kingdevnl]

So far I've been playing with Zinc and it's been working great. It would be cool if we could have a log system like Logtash for zinc. I've always mainly been using Elastic for a log system for multiple machines.

[wsw70]

@kingdevnl you may want to consider something like Fluend which makes data transformation IMO easier than LogStash (and is not bound to a product)

[susinths]

Can also mention Fluent Bit if you want a light weight forwarder written in C.

[Starttoaster]

IMO you should check out Loki. ElasticSearch wasn't made for logs anyway. ElasticSearch indexes everything which is why it has JVM memory pressure issues, whereas Loki only indexes labels associated with a logline.

[prabhatsharma]

Fluentd, Fluentbit and vector (Anything that can push to elasticsearch) should be able to push to zinc with some configuration (Authentication and URL) changes. I will create some docs to make it easy.

[kingdevnl]

I couldn't figure out how to handle auth with fluentd

[hugollanos]

Maybe a working fluentd configuration file example will be great to start working on it... have tried without success and no more ideas now

[susinths]

Here is at least a working Fluent Bit config:

[SERVICE]
    Flush        1
    Log_Level    debug
    Parsers_File parsers.conf
    Daemon Off

[INPUT]
    Name tail
    Parser 	syslog-rfc3164
    Path        /var/log/*
    Path_Key 	filename
[OUTPUT]
    Name  es
    Match *
    Path /es
    Index syslog
    Type  journal
    Host lb02.localdomain
    Port 4080
    Generate_ID On
    HTTP_User admin
    HTTP_Passwd secret

[FILTER]
    Name parser
    Match *
    Key_Name data
    Parser syslog-rfc3164

I've played around using Fluent Bit and it's systemd pluging to forward logs directly from systemd journal, but so far i've not been able to. The below is the config i've tried with:

[SERVICE]
    Flush        1
    Log_Level    debug
    Parsers_File parsers.conf
    Daemon Off
[INPUT]
    Name systemd
    Tag      *
    Path        /run/systemd/journal
[OUTPUT]
    Name  es
    Match *
    Path /es
    Index systemdlog
    Type  journal
    Host lb02.localdomain
    Port 4080
    Generate_ID On
    HTTP_User admin
    HTTP_Passwd secret

If anyone has any idea about what i am doing wrong, please let me know :)

[susinths]

Alright, the systemd plugin of Fluent Bit actually works when i am using it directly from the host OS and not running Fluent Bit as a container. This is really great. Next we need to Zinc in k8s/OpenShift, it would be awesome to get it working there :)

[hengfeiyang]

zinc has already supported filebeat in version v0.1.7.

[yourchanges]

A golang-implemented logstash https://github.com/childe/gohangout work with zinc, can save lots of resource

[hengfeiyang]

@yourchanges can you add an example of gohangout ?

[wsw70]

A golang-implemented logstash https://github.com/childe/gohangout work with zinc, can save lots of resource

It seems to be a nice product with a good documentation but it is unfortunately in Chinese, so limited to the speakers of Chinese.

[yourchanges]

@yourchanges can you add an example of gohangout ?

I will try it on next week.

[yourchanges]

@wsw70 here is an English readme file contains mostly doc. https://github.com/childe/gohangout/blob/master/README-EN.md

[yourchanges]

now with a gohangout zinc output plugin , gohangout + zinc works well on our PRD env, replace the ELK stack, simple and stable.

the gohangout zinc output plugin see: https://github.com/winwill2012/gohangout/commit/6d9102ef53d18426bacc373d00e098515a4dcbaa

[hengfeiyang]

@yourchanges can you try to create a merge request for https://github.com/childe/gohangout about zinc support ?

[yourchanges]

see https://github.com/childe/gohangout/pull/165

[yourchanges]

Since the gohangout owner reject the merge request, and suggest use the plugin ways, so pls use the following plugin: https://github.com/9ji/gohangout-output-zinc

[prabhatsharma]

Closing this issue. Look at https://github.com/zinclabs/zincsearch/issues/782 for explanation.