Metrics on workers

[akira]

Various metrics on worker performance (perhaps broken out by worker type):

• Frequency per second
• Average elapsed time
• Average wait time
• etc

This can be implemented as middleware. See Sidekiq Metrics for example.

Consider using: https://github.com/pinterest/elixometer

[j-mcnally]

Should we be storing this to redis, or just log it?

[akira]

Not sure yet. This is definitely more of a nice to have though. I was wondering if there's a coda hale metrics type library for erlang that can be configured to output to different sources.

[Gyllsdorff]

I think it would be enough to just expose the metric to callback somewhere and let the developer connect it to their prefered tool. For some people just want to log the data and some uses a full metric tool like datadog.

[mbuhot]

We've recently implemented a simple middleware to export counts and timings from Exq to StatsD via ExStatsD in the same format as sidekiq pro metrics

Would you be interested in a PR? It would add an optional dependency on the StatsD client package.

[LeeXGreen]

@mbuhot I'd be interested to see your middleware, even if it doesn't get merged into exq :)

[mbuhot]

@LeeXGreen Here's a gist: https://gist.github.com/mbuhot/93eab4472901eb18294d4eff1456aff6#file-exq_statsd_middleware-ex

[LeeXGreen]

@mbuhot Thanks! I was about to build this exact same thing, and this is exactly what I need.

One small question:

https://gist.github.com/mbuhot/93eab4472901eb18294d4eff1456aff6#file-exq_statsd_middleware-ex-L35 The code references "duration" in the pipeline assigns, and it appears that my pipelines do not have a "duration" value. Did you have another piece of middleware that sets this? I think I have a workaround for this, but I'd appreciate any info you remember.

Thanks again!

[mbuhot]

@LeeXGreen yes we also extracted the timing code out of the Logger middleware into its own middleware, which just populated the duration value. It's then used by the metrics middleware and a custom Logger middleware.