Expose metrics for monitoring

[sargun]

We need to expose metrics for monitoring. A service without metrics may as well not be running if it isn't monitored is the common mantra these days amongst SREs. The other aspect of this is that users must monitor the usage of a given mesos-dns instance in order to properly capacity plane.

Fortunately, Go has a standard package, expvar for exporting variables, and data via HTTP. Though, this package doesn't have great components to help users keep track of metrics. There are a few libraries to handle the collection of statistics:

• Go-metrics: https://github.com/rcrowley/go-metrics
• Armon's Go-metrics: https://github.com/armon/go-metrics
• Go-Kit's metrics package: https://github.com/go-kit/kit/tree/master/metrics

RCrowley's package seems the most popular one of the three that's used in many open source projects. On the other hand, Armon's library is used in anger at Hashicorp.

The go-kit library appeals to me the most, because it has integration with statsd, and expvar, allowing us to export metrics to DCOS via statsd, and other users via expvar.

The metrics I'm interested in capturing are as follows:

• Histograms:
  • Request latency
  • Upstream resolution latency
  • Requests
• Counts
  • Records

Other:

• Memory usage

[jdef]

FWIW kubernetes uses prometheus for exporting metrics.

https://github.com/prometheus/client_golang

If anyone is working on DCOS metrics it might be nice to pick a lib that will integrate well with that.

On Fri, Nov 13, 2015 at 11:57 PM, Sargun Dhillon [email protected] wrote:

We need to expose metrics for monitoring. A service without metrics may as well not be running if it isn't monitored is the common mantra these days amongst SREs. The other aspect of this is that users must monitor the usage of a given mesos-dns instance in order to properly capacity plane.

Fortunately, Go has a standard package, expvar https://golang.org/pkg/expvar/ for exporting variables, and data via HTTP. Though, this package doesn't have great components to help users keep track of metrics. There are a few libraries to handle the collection of statistics:

• Go-metrics: https://github.com/rcrowley/go-metrics
• Armon's Go-metrics: https://github.com/armon/go-metrics
• Go-Kit's metrics package: https://github.com/go-kit/kit/tree/master/metrics

RCrowley's package seems the most popular one of the three that's used in many open source projects. On the other hand, Armon's library is used in anger at Hashicorp.

The go-kit library appeals to me the most, because it has integration with statsd, and expvar, allowing us to export metrics to DCOS via statsd, and other users via expvar.

The metrics I'm interested in capturing are as follows:

Histograms:

• Request latency

  • Upstream resolution latency

  - Requests

  Counts

• Records

Other:

• Memory usage

— Reply to this email directly or view it on GitHub https://github.com/mesosphere/mesos-dns/issues/349.

[sargun]

Paging @nickbp, who is working on DCOS metrics.

[sargun]

@jdef It appears like go-kit's metrics support prometheus.

[panda87]

@sargun Hi, any update on that? We recently experience high latency for resolve dc/os services. Can you pls tell update when you think this feature will be released?

Thanks

[jdef]

AFAIK we've discussed statsd support for services running on DCOS. for core services within dcos, I don't think we've settled on a standard OSS API yet.

On Sun, Jan 15, 2017 at 2:14 AM, panda87 [email protected] wrote:

@sargun https://github.com/sargun Hi, any update on that? We recently experience high latency for resolve dc/os services. Can you pls tell update when you think this feature will be released?

Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mesosphere/mesos-dns/issues/349#issuecomment-272678576, or mute the thread https://github.com/notifications/unsubscribe-auth/ACPVLEjuLgLXPztkR9l8k_5Zemq8jOHbks5rScdKgaJpZM4GiSnq .