"default_index": "logstash-*" will open too many shards

[danielmotaleite]

i have about 2 month of logs stored and opening logtrail it complains (actually ES is the one that complains) about too many shards

I can workaround by using default_index": "logstash-2017.* but this also means that i will have to put some cron limiting the index name to the current month

Is it possible to define something like logstash-YYYY.MM.DD-*. where the number of days to query is set by the default_time_range_in_days

[soulis-siluos]

might be because when it queries for /logtrail/hosts it looks in all your indexes. i updated my server.js in the following way to get around this. in line 1 i have imported date format

var dateformat = require('dateformat'); and added the following method

/** * Takes a pattern of the form blah-* and converts it into a list of indexes with dates appended */ function toDatedIndexPattern(prefix, start, days, as) {

// alias
var self = this;

// sanity checks
if ((!prefix) || (!start) || (days < 0))
	return;

// remove the star at the end
prefix = prefix.replace('*','');

// default
as = (as) ? as : 'string';

// indexes we care about
var indexes = [];

// how many millis are in one day          
var DAY = 86400000;          

// add the number of days
for (var index=0; index<days; index++) {
	var date = new Date(start.getTime() + (index * DAY));
	
	// the 3rd arg 'true' ensures that we keep the date in UTC instead of local time
	var indexname = prefix + dateformat(date, 'yyyy.mm.dd', true);
	
	// add to set of indices
	indexes.push(indexname);
}

// return content
return as == 'string' ? indexes.join(',') : indexes;

}

then within the /logtrail/hosts route i modified it so that i use

  // how many millis are in one day          
  var DAY = 86400000;          
 
  // shrink down the number of indexes we will look at
  var start = new Date(new Date().getTime() - (DAY * selected_config.default_time_range_in_days));
  var indexes = toDatedIndexPattern(selected_config.es.default_index, start, selected_config.default_time_range_in_days, 'string');

then the hostAggRequest in the same method instead of var hostAggRequest = { index: index, ... i use var hostAggRequest = { index: indexes, ignore_unavailable:true,

[soulis-siluos]

OK so i managed to get a rolling window working for that query as well. Additional modifications are required.

1. update method $scope.seekAndSearch in app.js. to add rehost update when date changes ... setupHostsList(); $scope.hideDatePicker();

2. in function setupHostsList we need to add the seek time on where we are var params = { index: selected_index_config.es.default_index, }; if ($scope.pickedDateTime) params.seek = Date.create($scope.pickedDateTime).getTime(); ...

3. need to update the server side server.js insite the server.route('/logtrail/hosts') route

   // how many millis are in one day
   var DAY = 86400000;

   // seek region var start = null; var indexes = null; var seek = (request.query && request.query.seek) ? request.query.seek : null; if (seek) { // shrink down the number of indexes we will look at start = new Date(request.query.seek - Math.floor( (DAY * selected_config.default_time_range_in_days) / 2 ) ); indexes = toDatedIndexPattern(selected_config.es.default_index, start, selected_config.default_time_range_in_days+1, 'string'); } else {

     // shrink down the number of indexes we will look at
     start = new Date(new Date().getTime() - (DAY * selected_config.default_time_range_in_days));
     indexes = toDatedIndexPattern(selected_config.es.default_index, start, selected_config.default_time_range_in_days+1, 'string');
   

   }

And that should do it. it will look for indexes before and after the seek time for systems

Note : you may wish to look at https://github.com/sivasamyk/logtrail/issues/86 as well as it adjusts the hosts lookup when you change indexes in the settings

[soulis-siluos]

One last update. You will want to update the callWithRequest handler in this function as well to callWithRequest(request,'search',hostAggRequest).then(function (resp) { //console.log(resp);//.aggregations.hosts.buckets); reply({ ok: true, resp: (resp.aggregations && resp.aggregations.hosts) ? resp.aggregations.hosts.buckets : [] }); }).catch(function (resp) { if(resp.isBoom) { reply(resp); } else { console.error("Error while fetching hosts",resp); reply({ ok: false, resp: resp }); } });

That way if the aggs does not return any data, your UI will still update to show no fields correctly otherwise it would likely throw an error

[danielmotaleite]

@soulis-siluos any change to create a pull request with those changes, so that @sivasamyk can merge then?

[pdiniz13]

has this feature been added?

[t-beckmann]

@pdiniz13, does not look like this is in the repo. Could you contribute via a pull request?

[t-beckmann]

Providing a PR as https://github.com/sivasamyk/logtrail/pull/428 on behalf of the changes of @soulis-siluos above.