Scirius strange behaviour: No continuous data collection

[larsbru]

I have installed a fresh SELKS System an did the Update by script as described in SELKS Handbook.

My system collects only some hours per day... thats strange. Suricata, Elastic, Disk and Memory icons are green. Does someone know how to fix this?

[pevma]

Are the graphs in Kibana continuous or they show similar behavior?

[larsbru]

Yes, exactly the same. As if no traffic comes in in that specific period of time.

[pevma]

Can you check the log files themselves then and see if in there is a gap int the timestamps as well ? If not then most likely it is something related to shipping I guess.

[larsbru]

What logfile do you mean? I found some errors in e.g. "logstash-plain-2018...log" what other files shall i concentrate on?`

These errors are dropped very often (almost every 15minutes....[2018-05-07T00:15:29,123][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"$ [2.....

Maybe my config file is wrong?

I think shipping of the Data should be fine as i also used a virtual machine of Alienvault before to analyse the same datastream... My network traffic is simply mirrored to hat NIC "ens33" by the switch itself.

[pevma]

Can you describe a bit more about your set up - is it ELK5/6 ? The data i meant the data in the logfile that is getting shipped (eve.json) for example - is it continuous there ?

These errors are dropped very often (almost every 15minutes....[2018-05-07T00:15:29,123][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"$

Judging by the above I am guessing there is a problem with the logstash shipping - it has the pipeline stopped processing new events - seems logstash is not shipping. I would start troubleshooting form there.