SELKS
SELKS copied to clipboard
StamusNetworks
Problem with Elasticsearch after upgrade
Hi. I have a problem with GUI ES after upgrade. In System settings edition ont ES tab (Erase Elasticsearch data Clicking on the button will erase all Elasticsearch data except Kibana dashboards.) when hit button Clear I recieve those message:
CloseClearing failed: ES failure: ES transport error: 400 illegal_argument_exception {'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'Indices [.geoip_databases] use and access is reserved for system operations'}], 'type': 'illegal_argument_exception', 'reason': 'Indices [.geoip_databases] use and access is reserved for system operations'}, 'status': 400}
This is my selks_health_check
suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (running) since Fri 2021-10-01 02:04:42 EEST; 6h ago Docs: man:systemd-sysv-generator(8) Process: 18391 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 22 (limit: 4915) Memory: 2.2G CGroup: /system.slice/suricata.service └─18397 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash
Oct 01 02:04:41 ASOFTIPS systemd[1]: Starting LSB: Next Generation IDS/IPS... Oct 01 02:04:42 ASOFTIPS suricata[18391]: Starting suricata in IDS (af-packet) mode... done. Oct 01 02:04:42 ASOFTIPS systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-09-30 13:27:33 EEST; 19h ago Docs: https://www.elastic.co Main PID: 3729 (java) Tasks: 139 (limit: 4915) Memory: 7.3G CGroup: /system.slice/elasticsearch.service ├─3729 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=tr… └─3929 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Sep 30 13:27:19 ASOFTIPS systemd[1]: Starting Elasticsearch... Sep 30 13:27:33 ASOFTIPS systemd[1]: Started Elasticsearch. ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-09-30 12:08:41 EEST; 20h ago Main PID: 383 (java) Tasks: 61 (limit: 4915) Memory: 864.7M CGroup: /system.slice/logstash.service └─383 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC ->XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.…
Oct 01 03:47:07 ASOFTIPS logstash[383]: [2021-10-01T03:47:07,280][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 03:47:08 ASOFTIPS logstash[383]: [2021-10-01T03:47:08,003][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"} Oct 01 03:47:09 ASOFTIPS logstash[383]: [2021-10-01T03:47:09,277][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 03:47:25 ASOFTIPS logstash[383]: [2021-10-01T03:47:25,819][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 03:47:26 ASOFTIPS logstash[383]: [2021-10-01T03:47:25,820][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 03:47:26 ASOFTIPS logstash[383]: [2021-10-01T03:47:25,821][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 03:47:26 ASOFTIPS logstash[383]: [2021-10-01T03:47:25,823][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 03:47:26 ASOFTIPS logstash[383]: [2021-10-01T03:47:25,817][ERROR][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Attempted to sen… Oct 01 04:03:21 ASOFTIPS logstash[383]: [2021-10-01T04:02:54,600][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] Retrying failed … Oct 01 04:03:24 ASOFTIPS logstash[383]: [2021-10-01T04:03:21,270][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca21a05c69227f334d0c6198948f303fac6e50c03be43b13] R…est {:count=>1} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-09-30 12:08:42 EEST; 20h ago Docs: https://www.elastic.co Main PID: 722 (node) Tasks: 18 (limit: 4915) Memory: 372.0M CGroup: /system.slice/kibana.service ├─ 722 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid └─1148 /usr/share/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/share/kibana/src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/ki…
Sep 30 12:08:42 ASOFTIPS systemd[1]: Started Kibana. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-09-30 12:08:41 EEST; 20h ago Main PID: 385 (evebox) Tasks: 9 (limit: 4915) Memory: 4.4M CGroup: /system.slice/evebox.service └─385 /usr/bin/evebox server
Sep 30 12:09:22 ASOFTIPS evebox[385]: 2021-09-30 12:09:22 WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url … (os error 111) Sep 30 12:09:25 ASOFTIPS evebox[385]: 2021-09-30 12:09:25 WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url … (os error 111) Sep 30 12:09:28 ASOFTIPS evebox[385]: 2021-09-30 12:09:28 WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url … (os error 111) Sep 30 12:09:31 ASOFTIPS evebox[385]: 2021-09-30 12:09:31 WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url … (os error 111) Sep 30 12:09:34 ASOFTIPS evebox[385]: 2021-09-30 12:09:34 WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url … (os error 111) Sep 30 12:09:37 ASOFTIPS evebox[385]: 2021-09-30 12:09:37 WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url … (os error 111) Sep 30 12:09:41 ASOFTIPS evebox[385]: 2021-09-30 12:09:41 INFO evebox::server::main: Found Elasticsearch version 7.15.0 at http://localhost:9200 Sep 30 12:09:41 ASOFTIPS evebox[385]: 2021-09-30 12:09:41 INFO evebox::server::main: Starting server on 127.0.0.1:5636, tls=false Sep 30 12:11:12 ASOFTIPS evebox[385]: 2021-09-30 12:11:12 INFO evebox::server::main: Creating anonymous session for user from 127.0.0.1 with name asoftplus Sep 30 12:11:12 ASOFTIPS evebox[385]: 2021-09-30 12:11:12 WARN evebox::elastic::eventstore: Elasticsearch response has no aggregations Hint: Some lines were ellipsized, use -l to show in full. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-09-30 12:08:42 EEST; 20h ago Main PID: 711 (sh) Tasks: 12 (limit: 4915) Memory: 21.3M CGroup: /system.slice/molochviewer-selks.service ├─711 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 └─713 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini
Sep 30 12:08:42 ASOFTIPS systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2021-09-30 12:14:43 EEST; 20h ago Process: 1877 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 1877 (code=exited, status=1/FAILURE)
Sep 30 12:13:13 ASOFTIPS systemd[1]: molochpcapread-selks.service: Main process exited, code=exited, status=1/FAILURE Sep 30 12:13:13 ASOFTIPS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Sep 30 12:14:43 ASOFTIPS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Sep 30 12:14:43 ASOFTIPS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4. Sep 30 12:14:43 ASOFTIPS systemd[1]: Stopped Moloch Pcap Read. Sep 30 12:14:43 ASOFTIPS systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Sep 30 12:14:43 ASOFTIPS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Sep 30 12:14:43 ASOFTIPS systemd[1]: Failed to start Moloch Pcap Read. scirius RUNNING pid 6663, uptime 17:05:57 ii elasticsearch 7.15.0 amd64 Distributed RESTful search engine built for the cloud ii elasticsearch-curator 5.8.4 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.14.0 amd64 no description given ii kibana 7.15.0 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates. ii logstash 1:7.15.0-1 amd64 An extensible logging pipeline ii moloch 3.0.0-1 amd64 Moloch Full Packet System ii scirius 3.7.0-6 amd64 Django application to manage Suricata ruleset ii suricata 1:2021090701-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 5.9G 0 5.9G 0% /dev tmpfs tmpfs 1.2G 8.5M 1.2G 1% /run /dev/sda1 ext4 28G 8.0G 19G 31% / tmpfs tmpfs 5.9G 0 5.9G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup /dev/sda7 ext4 3.5G 15M 3.3G 1% /home /dev/sda6 ext4 60G 16G 41G 28% /var tmpfs tmpfs 792M 0 792M 0% /run/user/1001
And I have another issue with suricata logrotate - log is rotated but files in /var/log/suricata/StatsByDate/ keep growing and I need to manualy delete it every 2 days or i recieve database is full . I don't know how to manage it properly.
This is my suricata config
/var/log/suricata/eve.json { daily rotate 1 olddir /var/log/suricata/StatsByDate/ compress missingok notifempty dateext postrotate /bin/kill -HUP $(cat /var/run/suricata.pid) endscript }
Thank you for your report. I can reproduce the first error - ES clear data, that seems like a bug that we will tackle. The second issue - how big does it grow ?
For 2 days this disk is completly full: /dev/sda6 ext4 60G 16G 41G 28% /var And another bug is in opt/selks/delete-old-logs.sh
- delete_indices need to be replaced with delete-indices in other case delete-old-logs.sh dont work and Elasticsearch logs also full the disk.
I was abble fix this one bug but main problem with it this bug is that is default settings when someone hit Y trought upgrade process.
I think that could be fixed in future version and will be cool.
p.p. in /var/log/suricata eve.json is 10GB also ...
Yes sure - that will be fixed - thank you for the report! Just as a suggestion - you might also consider bigger disk if it gets full inside 48hrs.
Thanks for fast response. I know that I can add more disk space. Is it wrong if I make crontab job like this? find /var/log/suricata/StatsByDate/* -type f -mtime +1 -delete
And another one question. Is eve.json is cleared through curator?
No problem! You can setup the crontab anyway you would like.
eve json is rotated/logrotated , not handled by curator/