weaveworks
Review the API
Look at the current API. Make sure it is fit for purpose regarding:
- tracing
- metrics
- logging
Check on the state of testing.
We have been assured that the other team are not looking at things like this (although I honestly don't know), so hopefully there wont be any conflict.
You can focus just on areas we care about for the observability track at first if you like.
Outcome can be either "do the thing" or spin off stories based on your recommendation.
We have basic request logging in our middleware, but it would be great to add it to the core/server handlers. No metrics or tracing yet.
I'm going to start with some tasteful logging. I think I will also add a way to configure the log level, rather than just having it be on/off. More notes to follow
Note to self: while doing other things, I have noticed that the things supposedly logged by appConfig.Logger do not appear anywhere. Find out if I am doing something wrong or if it is simply not wired properly.
turns out the -l flag vanished from the Tiltfile somewhere along the way.
Putting this down for a minute, because honestly I need to think of something else for a bit.
My next steps are (if anyone want to pick up one of these, feel free. otherwise i will come back to them):
- [ ] Looking at how we can improve the GRPC logging middleware to get more informative messages (and not just on failure)
- [ ] Considering which metrics (if any) we would want to emit at this time
- [ ] Play around with jaeger tracing middleware (or any other alternative) and see how it could be useful to us
- [ ] Have a review of all tests of core and other relied-upon components and come up with a list of where we are lacking behaviour checks. Probs produce some tickets out of this.
This has been partially addressed by some middleware we've added recently for Prometheus metrics. Traces and logs still need to be reviewed.
