heroku-datadog-drain-golang icon indicating copy to clipboard operation
heroku-datadog-drain-golang copied to clipboard

Published 20 hours ago • verified publisher icon apiaryio

heroku.router.request.* anomalies?

Open mezis opened this issue 8 years ago • 5 comments

Hi all!

Are there know issues with the heroku.router.request.* metrics? We're seeing counts that are completely off (90% of the volume is missing in action), and the median/average connect/service time don't seem to add up with Heroku's own metrics dashboard, either. This is starting to smell like parsing issues (ignoring many log lines) but I'm not having luck spotting the issue so far.

mezis avatar Dec 01 '16 17:12 mezis

I am not aware of any, there was an issue with the sample rate, but that's now 1.0 so it should not drop anything. For the parsing issues there might be some info in the logs https://github.com/apiaryio/heroku-datadog-drain-golang/blob/master/client.go#L91 or in https://github.com/apiaryio/heroku-datadog-drain-golang/blob/master/logproc.go#L54 have you seen any of those?

w-vi avatar Dec 01 '16 17:12 w-vi

Nothing comes out at INFO or WARN levels, no. And DEBUG level isn't too useful at our request volumes (north of 120k req/min) - simply too much noise. I'll keep digging, and report here if I manage to figure it out :/

mezis avatar Dec 01 '16 17:12 mezis

AFAIK the delivery is done over UDP so maybe there is a big packet loss.

w-vi avatar Dec 01 '16 18:12 w-vi

maybe this is related https://github.com/DataDog/datadog-go/issues/20

w-vi avatar Dec 01 '16 18:12 w-vi

Is this still valid?

abtris avatar Feb 03 '20 11:02 abtris