consider supporting local-core metrics

[siddontang]

Now using metrics across multi threads has a low performance, so we have to use local metric and then flush to the global metric sometimes, it is not convenient, maybe we can use core-local metrics, But I don't know whether it is easy to do or not.

See http://rocksdb.org/blog/2017/05/14/core-local-stats.html

[jiacai2050]

Anyone working on this? I would like to contribute as my project depend heavily on this.

[mxinden]

As far as I know there is no-one working on this @jiacai2050. Before you do, would you mind describing your use-case?

This crate supports local metrics via the local module which operates on non-thread-synchronized memory, thus bypassing the performance cost of atomic operations, eventually being flushed back to their global thread-synchronized counterparts.

I am not aware of a use-case that can not be achieved with the local module, but only with the currently missing idea of metrics local to a given core or thread.

[breezewish]

@jiacai2050 I think you can try with thread local metrics + static metrics. According to our experiments this is already very efficient, although theoretically core-local is even better than thread local :)

[jiacai2050]

@mxinden @breeswish Thanks for your suggestion. local module is useful for when update metric in for-loop, but what I want is thread-local metrics update in its own thread, and only flush when prometheus scrapes. AFAIK, thread-local metrics is better than core-local which requires flush when return from functions. I will check what @breeswish mentions, see if it will meet my requirements.

[breezewish]

@jiacai2050 The current implementation of the thread local metrics provided by this crate is an auto-flush one. Every time you update a thread local metric, the thread local last update time is checked, and then the thread local metric is synced to the global one if specified interval is elapsed. For both public Push and Pull interfaces, the value in the global one is used.

This kind of thread local metric avoids frequently updating a global shared memory, i.e. reduces contention to improve performance. It is mostly useful if you update your metrics very frequently in multiple threads (e.g. 1000 times per sec, updated by multiple worker threads).

Static metrics macros in this crate is another story. Normally when you use metrics with labels (i.e. metrics vectors), the label needs to be put or retrieve from a HashMap in runtime. However, in many scenarios the metric labels are deterministic and known at compile time. Static metrics macros help you define these static metrics easily, in a way similar to normal metric vectors, without the need to write many boilerplates. It saves HashMap lookup cost.

The two techniques (static metrics and thread-local metrics) can be used together to achieve the best performance this crate offers.