census-instrumentation
Provide best practices for metric instrumentation
Currently there does not seem to be comprehensive reference documentation on how to use OpenCensus. The language specific guides do not seem enough to cover this.
Packages like ochttp give away hints that there's such a thing as namespacing by domain names and / separated paths. But rules on how to use them are missing. In particular the example on a custom metrics just uses a flat view name, which causes some confusion.
For example, what's new to me (compared to Prometheus) is inclusion of available tags into the name (..._by_hostpath), which raises some questions I'd need guidance on. I'd generally expect http metrics to have status, method, path, and possibly host. Would my metric then be http_requests_by_hostpathmethodstatus, http_requests_by_host_path_method_status or similar?
Similarly, information on recommended usage of tags and their cardinality seems very important.
Based on past experience, people will not get those things right by default and it can take years for initial mistakes to get fixed and rolled out. There's probably no single right way, which makes it even more important to come up with best practices early on so people experience as much consistency as possible across systems.
I mostly ran into this w.r.t metrics, but the same probably applies for traces.
For inspiration/reference, this is the relevant Prometheus documentation:
- https://prometheus.io/docs/practices/naming/
- https://prometheus.io/docs/practices/instrumentation/
Go's ochttp also seems to provide different views by default for the same measure, rather than exporting at the maximum granularity and leaving aggregation to the the backend.
I can see this having certain benefits, but an official motivation and recommendations on when and how to use it would help here as well.
The default HTTP views are really not very useful IMO. We deliberately exclude all tags that even potentially have high cardinality, erring on the side of safety. I think almost all users will want to define their own fine-grained views that include things like path and host tags.
On the more general point of providing guidelines on what how to create measures, views and tags I absolutely agree that we need more guidance. I think the situation is worse with the stats package than tracing because it's more unusual and there are more concepts in play.
The default HTTP views are really not very useful IMO.
This is very confusing given we suggest our users to register to these views given they don't always know what to collect from HTTP requests. If the default HTTP views are not useful, then why do we have them in the ochttp package in the first place?
I think they were initially useful when they included the path and the host header. But since we had to remove those over cardinality concerns I don't think they are the most useful way to instrument most HTTP servers (at least the kinds that I have worked on in the past).
If through application-specific knowledge, you know that the path and host header will be low cardinality (as is the case most of the time) then I would probably recommend creating views that include these tags.
I would love to be able to provide useful HTTP views by default. But I just don't see how we can do that without making stronger assumptions about the application. I think we do our users a disservice by saying "these are the essential views you should install" because then they might not go through the process of thinking about what their application really needs.