ElasticSearch Support

[etishor]

This issue is for tracking status and ideas related to ElasticSearch ecosystem.

The first bits for elasticsearch integration are available on dev & master branches and are also pushed to latest nuget v0.2.13.

Most of the code is in the ElasticSearchReport.cs. Turns out reporting metrics to ES is very straight forward.

~~At the moment only gauges, counters and meters are reported but Histograms & Timers should be trivial to add (hopefully today).~~

[etishor]

@kadaan Related to logstash - i'm not familiar with it, but if i'm reading the docs correctly, it should be possible for logstash to grab the JSON metrics from the http endpoint exposed by the metrics library.

I'm not sure if that is how the logstash - ES is supposed to work, but if that was possible it would simplify the integration a lot.

[etishor]

As this functionality does not impact the stable metrics functionality it is also included in the latest .nuget for easier testing.

[kadaan]

Yup, that is how it works. The main thing that would be good it to ensure the current json form is "ideal" and to create an explicit ES mapping to describe the data.

On Dec 14, 2014, at 4:23 AM, Iulian Margarintescu [email protected] wrote:

As this functionality does not impact the stable metrics functionality it is also included in the latest .nuget for easier testing.

— Reply to this email directly or view it on GitHub.

[ShawInnes]

This is great functionality - however if the ES server is not running it will cause an exception. What are your thoughts about putting a try-catch around the WebClient code? Is it considered safer to swallow the exception or force the end-user to deal with the exception?

[etishor]

ES bits are still in very early stages. Exception handling was not a concern yet. It would normally be delegated to MetricsErrorHandler where the end-user has the possibility to handle the error, but its not required.

[robertaves]

Looking for an functionality to index documents daily so we can drop indices rather then delete documents. For example, when we use "metrics" as an index, the reporter would post to index "metrics-2015.06.10" if the time-stamp of the samples was on June 10th 2015 This allows elasicsearch's Curator to drop indexes after a configured period and saves from massive indexes.

Does this sound feasible?

[etishor]

It does sound feasible. I'm not using ES at the moment so my focus will not be on this at the moment.

I know ES supports TTL for documents and they get automatically deleted when expired. Maybe that would be a better approach.

In the near future I plan to re-iterate over the ES support, but not until the HDR histogram bits are finalized.

[kadaan]

TTL on documents is not recommended.

[etishor]

I did not know that. Any references?

[alkampfergit]

I think @kadaan refers to giving a ttl for each distinct document, you can setup a TTL for the whole index https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-indices.html#indices-ttl,

I'd like to be able to set ttl for distinct type of metrics. In real situation there are metrics I do not care to keep around for lots of time, but some important metrics can last forever.

You should also add some default mapping instead of letting elastic-search guessing everything, as an example Name, Type, Unit for gauges should be not-analyzed. I think that for metrics, all fields of type strings should be set "not-analyzed" by default.

[kadaan]

Why not just write the events out to JSON on file and use log stash to ingest them?

[kadaan]

@alkampfergit that TTL you referred to does not sound like it deals with deletion of documents. It sounds more like the poll interval for the index actually purging documents whose TTL has expired.

AFAIK the only three ways to delete documents in elasticsearch are to set a TTL on the document, to delete an index, or to run to delete by query.

[alkampfergit]

According to the documentation (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)

You can provide a per index/type default _ttl value as follows:

{ "tweet" : { "_ttl" : { "enabled" : true, "default" : "1d" } } } In this case, if you don’t provide a _ttl value in your query or in the _source all tweets will have a _ttl of one day.

Using json file and logstash is probably the preferred way to send log to elasticsearch, but it should be an option, because it introduce a little bit of extra complexity in setup.

There are also need to do some "mainteinance of the log". If I have a gauge, and configured to send log to ES each 2 seconds, probably I'd like to have full resolution for the last 24 hours, then consolidate last X days to have 1 value for the gauge for each minute (averaging all the logs in each minute), then probably 1 value for each hour for the last year.

In this way i can have different resolution for a gauge depending on how much the log is old.

[robertaves]

Also can you add the server name to the metric. We have a cluster of machines doing the same thing and it would be nice to aggregate by server ( name or ip )

[Stiek]

I am currently implementing Metrics.NET in a Proof of Concept project. I am sending the metrics to ElasticSearch with the following config: Metric.Config .WithReporting(report => report.WithElasticSearch("localhost", 9200, "metrics", TimeSpan.FromSeconds(1))) .WithReporting(report => report.WithCSVReports(@"D:\Temp\reports", TimeSpan.FromSeconds(1)));

In my .csv report directory I see data for all metrics used: Counter, Meter, Histogram and Timer. In ElasticSearch, using Head, i see Meters and Counter data, but no Histogram and Timer data.

I am using the Metrics.NET NuGet package version 0.3.3-pre (Prerelease).

Can you help me figure out why I am not seeing Histogram and Timer data in ElasticSearch?

EDIT - Found the problem: https://github.com/elastic/elasticsearch/issues/14594 "Starting with Elasticsearch 2.0 field names can not contain the . character anymore." Since both Timer and Histogram have the field [Percentile 99.9%], adding these documents into ElasticSearch will fail and no Histogram or Timer data will ever be entered into ElasticSearch. Also see: https://github.com/etishor/Metrics.NET/issues/113

[emulk]

Hello,

[Field name [Percentile 99.9%] cannot contain '.'] i'm having the same issue right now, how can i solve it ?

I'm trying to use Metrics.net with elasticsearch2.2.0 and grafana.

Is possible to change the Metrics.net column name ?

Thank you