Switch to hourly logstash indices?

[shartge]

After fixing the "curator does not delete" issue I had another look at the amount of data that is pushed into elasticsearch. On my system this averages at about 30GB per 24 hours, so I increased /sexilog to 120GB and set the curator cutoff at 100GB, giving me about 3 days worth of information.

But when curator kicks in, this will delete a big chunk of data, a complete day.

I propose to switch to hourly indices which will result in much smoother data deletion.

[rschitz]

I definitively will look into this but i think it's not possible in the present version

Thanks again for your feedback

Can you mail me your configuration ?

[shartge]

I have nothing to show at the current time, because I have not yet reconfigured anything because I have currently no access to my sexilog instance. I will look into this further in the following week, but from a quick glance at the logstash and kibana documentation I gather this is quite straight forward to implement:

1. add index => "logstash-%{+YYYY.MM.dd.HH}" to the elasticsearch output in logstash
2. reconfigure the index to use in the dashboard configuration of kibana
3. Profit.

[sexibytes]

Yes i get it now, thanks again.

[rschitz]

More indices means more shard so we must look into the IO impact too

[shartge]

Does it really increase the IO that much? Time series data is written only once and then only touched by reading. Having to read/sift through 30GB of data out of one index or 24 indices for 24 hours shouldn't matter that much, I think.

[rschitz]

we'll ask to elastic.co guys in the next meetup, i'll let you know. thanks a again for your involvement Sven ;)

[shartge]

I have been running with hourly indices since I created this issue ticket and have so far not found any apparent performance problems.

But I had problems with curator not deleting any indices and elasticsearch throwing an exception:

[2015-04-24 16:50:54,396][WARN ][http.netty               ] [Ectokid] Caught exception while handling client http traffic, closing connection [id: 0x485f4c9f, /127.0.
0.1:50382 => /127.0.0.1:9200]
org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException: An HTTP line is larger than 4096 bytes.

This is because of https://github.com/elastic/elasticsearch/issues/1174

The solution is to increase the initial line length via

http.max_initial_line_length: 64k

in /etc/elasticsearch/elasticsearch.yml

[rschitz]

Thanks Sven, we also need to change all the sexiboards https://gist.github.com/sexilog/ We also ran ES 1.5 on demo.sexilog.fr without any issues so far