Sampling support

[hagen1778]

Metrics update in application's hot path may be expensive. Some heavy-loaded web servers or processing pipelines usually serve hundreds of thousands of events per second, and to have the visibility they also increment a bunch of metrics for each event. It would be cool, if metrics package could have native sampling support. For example:

requestsTotal = metrics.NewCounterSampled("requests_total", 0.1)

Every call to requestsTotal.Inc() will have 10% chance to actually increment the counter, and 90% chance to do nothing. But when increment actually happens, it will be +=10 instead of +=1.

Workaround. Batch metrics updates in a hot path and make them periodic (e.g. every 100ms) with a fixed rate.

[valyala]

It would be great to have performance numbers and/or cpu profiles obtained from highly loaded systems where metrics sampling could result in better performance.