tracing

tracing copied to clipboard

I recently started using tracing in one of my projects, and one of the current major pain points is that I seem to be having an incomplete mental model of how the subscriber/layer/filter/registry functionality is working when configuring where tracing events are sent to.

This resulted that, when trying to set up tracing to sent events both to the standard formatter but with configurable log levels based on envfilter, and to sentry, in a rather painful setup process that cost multiple hours of essentially random guesses.

To improve this experience I think it would be very useful to have some documentation that explains the model used in the subscriber/layer/filter/registry system and how each of these concepts translates into parts of the eventual output architecture. In particular, it would be useful for this documentation to explain in detail how layers, filters, registries and subscribers compose, and what the effect of various ways of composition are (especially with regard to the impact of filtering).

I am willing to provide assistance in writing such a part of the documentation, especially around proofreading as an inexperienced user. However, that same inexperience means that I am currently not capable enough to make a first draft for this.

The tracing_subscriber::layer module has documentation that's intended to provide an introduction to these concepts. I'd love any feedback on what's currently missing from this documentation from your perspective, because we can definitely add to it.

I have an early draft of some documentation that explains the internals of tracing/tracing-subscriber from a high-level perspective: https://github.com/jswrenn/tracing-book/blob/main/src/internals.md

Feedback on layer module documentation

I have read the tracing_subscriber::layer module level documentation, and that was quite enlightening. However, discoverability is a problem. This is pretty useful documentation, but currently hidden in a sub-module of the tracing-subscriber package, which may not (and in my opinion likely is not) a place most beginners will think to look.

Also, there are still several smaller gaps that this documentation doesn't seem to cover: The filtering documentation is still somewhat vague, the global filtering says that global filtering goes from the outermost layer to the innermost, but the documentation is not that explicit which is the outermost layer. Furthermore, it seems to indicate that any layer filtering the event will cause all layers to ignore it, but that would suggest order doesn't really matter, which is not the feeling I get from the rest of the documentation.

The role of registry is still quite unclear after reading this. The documentation states that it is required to do per-layer filter, but other than that its role is somewhat vague, my understanding after reading this all is that it does something for spans, but that doesn't for me describe why it is a full subscriber even though it doesn't really handle anything.

Also, the difference between with, with_filter and and_then is not really all that clear. With_filter is most clearly different from the others as it deals with per-layer filters, but I'm not sure exactly why there exists both and_then and with, and what the uses are of either.

Finally, more of a minor nitpick, but the order of the documentation there could use some improving. Currently the documentation starts with how to compose layers, and then immediately moves to describing how to create a structure that might vary at runtime. I would suggest moving this second part on composing to later in the document, as it is not as important on a first read, but is quite large and cumbersome. Instead, most first timers would be better suited by starting the explanation of filtering (one of the main things you need to keep track of when combining layers) to earlier in the document.

General thoughts

Given the deeper dive I have done over the past few days I'm beginning to think that the problem might not entirely be with the documentation. It also seems to me that the API itself is quite complicated and could perhaps profit from a simplification.

As an example, the interactions between global filtering and per-layer filtering are quite complex. From what I got from the per-layer-filtering code, it seems that the per layer filter does still affect global filtering, in that any events filtered globally by the inner layer and also by the filter are no longer globally filtered by the result of with_filter, but anything that is allowed by the filter can still be globally filtered by the inner layer. Not sure how to simplify it, and in fact, doing anything about this and other complexities may just require a fairly large redesign of the entire stack.

PS: Apologies for the (somewhat) large delay in responding, it took me a bit to really dive into the layer module documentation and collect my thoughts.