go-carbon
go-carbon copied to clipboard
go-graphite
fetch failed no metrics found should not be an Error
we have 12 sites with local data written to each site. Data is in the format
Just for sake of clarity. Are you using carbonserver for queries of something else?
Yes we are. We had been using a graphite stack and did not get these messages in our logs. When we moved to go-carbon we started getting the errors in the logs. Since the return value is 400 graphite, and now carbonapi, also log this as an error. To mimic the behavior of graphite this should not be logging at all but I think changing this to a warning and not returning a 400 would be acceptable. go-carbon.conf.txt
I agree that 404 for absent metric shouldn't be rendered as error. @grzkv @gksinghjsr what is your take on this?
Meanwhile you can disable access logs and keep the rest of logging. Below is how to do this [[logging]] logger = "" file = "/var/log/go-carbon/go-carbon.log" level = "info" encoding = "json" encoding-time = "iso8601" encoding-duration = "seconds"
[[logging]] logger = "access" file = none level = "info" encoding = "json" encoding-time = "iso8601" encoding-duration = "seconds"
[[logging]] logger = "stat" file = "/var/log/go-carbon/go-carbon.log" level = "error" encoding = "json" encoding-time = "iso8601" encoding-duration = "seconds"
Does this work for you?
Yes, this does work for me and it looks like I should be able to figure out how to stop the 404 errors in the carbonapi logs using the same idea. Thanks very much.
You might consider trying zippet+api versions after this commit which contains the TLD filtering logic for zipper. It limits amount of go-carbon storages patrticular query will hit by learning from stores what top level metrics existing there and keeping this knowledge in a zipper cache.
@fgleason or you can use go-graphite/carbonapi which had the feature (TLD filtering logic and top-level cache) mentioned by @azhiltsov since several years ago. And current version of go-graphtie/carbonapi will treat not-found as "200" and empty response.
I'm also now working on error handling improvements in separate branch: https://github.com/go-graphite/carbonapi/tree/zipper_improve_errorhandling which makes this behavior more flexible and configurable (it allows you to override the status code that will be logged and returned)
