Jas
Jas
changed the topic name to 'filebeat' and i see the logs coming into kafka kafkauser@e71f29748d57:~/scripts$ /opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic filebeat --from-beginning but still noting in kibana.
Also read this topic: https://github.com/Cyb3rWard0g/HELK/issues/370 But the logs from filebeat dont appear under indexme-*
when i replayed some of the data using kafkacat manually it showed up in indexme-* root@helk:~# tail -50 /opt/bro/logs/current/conn.log > file.json root@helk:~# kafkacat -b KAFKA-IP:9092 -t zeek -P -l file.json...
root@helk:~# filebeat test output -e -c /etc/filebeat/filebeat.yml 2020-12-22T14:09:14.737Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] 2020-12-22T14:09:14.737Z INFO instance/beat.go:653 Beat ID: 8fef759b-78c5-4a26-be87-7e27816225e2 Kafka: 127.0.0.1:9092......
root@helk:~# filebeat -e -c /etc/filebeat/filebeat.yml [...trimmed...] /usr/local/go/src/runtime/asm_amd64.s:1373 2020-12-22T16:15:32.314Z ERROR [kafka] kafka/client.go:147 Dropping event: key not found github.com/elastic/beats/v7/libbeat/common.init /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:41 runtime.doInit /usr/local/go/src/runtime/proc.go:5474 [...trimed...]
root@helk:~# systemctl status filebeat ● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch. Loaded: loaded (/lib/systemd/system/filebeat.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2020-12-22...
output.kafka: enabled: true hosts: ["127.0.0.1:9092"] topic: filebeat compression: gzip max_message_bytes: 1000000
That error of "Dropping event: key not found" I had fixed. Forgot to post that. But funny thing, My last comment above was 9 hours ago. I did not touch...
Reading https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/0006-kafka-zeek-input.conf , I was assuming if I used the topic as zeek, then the zeek logstash parser would kick in. No ?
No it’s written in indexme-* and that’s what I am trying to fix now. Jasmeet On Wed, Dec 23, 2020 at 2:50 AM priamai wrote: > That is correct @jsinix...