disk full

[githubuser00]

After root(/) was full I moved /sexilog to another disk and now I am not getting proper results in http://192.168.111.101/index.html#/dashboard/elasticsearch/SexiBoard:msg

it displays very little info which is almost 24hrs old and anything before 24hrs is not there.

I am getting upto date email alerts from riemann but seems like kibana is broken.

root@test-sexilog:/etc/elasticsearch# cat elasticsearch.yml|grep -v '#'|grep -v "^$"
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100
threadpool.index.type: fixed
threadpool.index.size: 60
threadpool.index.queue_size: 200
index.translog.flush_threshold_ops: 50000
cluster.name: sexilog
index.number_of_shards: 1
index.number_of_replicas: 0
path.data: /sexilog
bootstrap.mlockall: true
discovery.zen.ping.multicast.enabled: false
indices.memory.index_buffer_size: 50%

root@test-sexilog:~# curl http://192.168.111.101:9200/_cat/shards      
kibana-int          0 p STARTED       24 177.2kb 192.168.111.101 Alexander Bont 
logstash-2017.11.14 0 p STARTED 34803750  23.7gb 192.168.111.101 Alexander Bont 
root@test-sexilog:~# 

What do I need to do start seeing latest messages in http://192.168.111.101/index.html#/dashboard/elasticsearch/SexiBoard:msg

[vmdude]

Root partition (/) and SexiLog partition (/sexilog) are already on 2 separate drives. /sexilog is on a dedicated 50GB drive by default.

Could you paste df -h results?

[githubuser00]

root@test-sexilog:~# df -h
Filesystem                                              Size  Used Avail Use% Mounted on
rootfs                                                  7.6G  5.5G  1.7G  77% /
udev                                                     10M     0   10M   0% /dev
tmpfs                                                   6.4G  288K  6.4G   1% /run
/dev/disk/by-uuid/4f5720f5-e552-494d-97f6-be61e0c30065  7.6G  5.5G  1.7G  77% /
tmpfs                                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                                    13G     0   13G   0% /run/shm
/dev/sda4                                               7.9G  5.0G  2.6G  66% /var
/dev/sdb1                                                69G   41G   25G  63% /sexilog
/dev/sdc1                                               493G  175G  294G  38% /dump
root@test-sexilog:~# 
root@test-sexilog:~# date;find /sexilog/sexilog -mtime 0|wc -l
Tue Nov 14 21:35:15 EST 2017
587
root@test-sexilog:~# date;find /sexilog/sexilog -mtime 0|wc -l
Tue Nov 14 21:35:24 EST 2017
590
root@test-sexilog:~# find /sexilog -mtime 0|head -10
/sexilog
/sexilog/sexilog/nodes/0/indices
/sexilog/sexilog/nodes/0/indices/kibana-int/0/index
/sexilog/sexilog/nodes/0/indices/kibana-int/0/index/segments_1d
/sexilog/sexilog/nodes/0/indices/kibana-int/0/index/segments.gen
/sexilog/sexilog/nodes/0/indices/kibana-int/0/translog
/sexilog/sexilog/nodes/0/indices/kibana-int/0/translog/translog-1426034847081
/sexilog/sexilog/nodes/0/indices/kibana-int/0/_state
/sexilog/sexilog/nodes/0/indices/kibana-int/0/_state/state-118
/sexilog/sexilog/nodes/0/indices/logstash-2017.11.15
root@test-sexilog:~# du -sh /sexilog/sexilog/
27G     /sexilog/sexilog/
root@test-sexilog:~# 

[githubuser00]

df_h.txt

[vmdude]

It seems you have free space on all partition. When you say you have only 24h old data, does it increase with time (I mean, have you got more than 24h now ?).

Could you check your elasticsearch cluster status, it's available from the head plugin: http://your_appliance_fqdn_or_ipv4/_plugin/head

[githubuser00]

Contents in that directory is always changing.

df_h2.txt

[vmdude]

OK, your cluster seems OK, indice have 35+ million documents (aka ESX log), so it seems everything is fine.

[githubuser00]

So it's not showing in the kibana but I do get up to date email alerts on those events.

[githubuser00]

Here is another screenshot for 30days:

[githubuser00]

next day (24hrs later):

[vmdude]

What were the exact operations you did when you said:

After root(/) was full I moved /sexilog to another disk

You may need to deploy a new appliance from scratch, it'll easier I think.

[githubuser00]

I was trying to avoid rebuilding because I am doing too many other things from the same system -therefore I was hoping to just fix the kibana.

[githubuser00]

maybe I can clear this out:

[vmdude]

Redeploying a new appliance from scratch with the same name and IP will take 10 minutes max although understanding exactly what has been done on your appliance in order to fix this can take a lot longer... I advise you to redeploy a new appliance if you don't have much time.

[githubuser00]

I think this is the why kibana has only previous days data to display: logstash-YYYY.MM.DD.txt

and here is the full directory long listing: find_sexilog_dir.txt