Very WIP: Composable log filters

[c42f]

It strikes me that the current design of AbstractLogger is very sink-centric; every logger is a sink, and one cannot pass around log filter pipelines as standalone objects. There seems to be a very close analogy with iterators which are very source-centric (one can wrap an iterator-filter around an source iterator, but not naturally compose several iterator filters together).

For iterators, many problems of composition and efficiency are elegantly solved by moving to transducers. Can we benefit from many of the same ideas here?

@tkf @oxinabox — I thought I'd write a quick note here to let you know a slightly different direction I've considered for log routing. It's likely trying to solve the same problems as LoggingExtras, but take a bit of a different tack. The code here is just a quick WIP from a while back and I doubt it takes the ideas from transducers very seriously, so take it FWIW.

Largely, I'd just like to stimulate the discussion a bit :-)

[tkf]

This is interesting. So, AbstractLogFilter is kind of a mapping logger -> logger′. We can make it literally so by defining (lxf::AbstractLogFilter)(logger::AbstractLogger) = FilteringLogger(logger, lxf). I'd actually generalize it and call it a "logger transform" (lxf) because there can be non-filtering logger transforms (e.g., LoggingExtras.TransformerLogger). Note that logger transform is not log event transform (which is what TransformerLogger does); it's a logger that is transformed.

Somewhat fun fact is, like transducers (which are "reducing function transforms"), ∘ would be "flipped"; i.e.,

lxf = LogLevelFilter(Error) ∘ MaxLogFilter()

global_logger(lxf(global_logger()))

first filters based on the log level and then looks up/stores the id in message_limits. ...or more confusing (but equivalent) one-linear is:

global_logger() |> MaxLogFilter() |> LogLevelFilter(Error) |> global_logger
# log event goes right to left

I think one catch might be that DemuxLogger and alike cannot be implemented this way (or more like this formalism does not clarify the implementation/API) since it has to know multiple "loggers." Similar concepts in Transducers.jl are "Zip" (which is probably not the right name https://github.com/tkf/Transducers.jl/issues/40) and GroupBy. They are unusual in the sense that they are transducers (or more precisely transducer factories) that take transducers and/or reducing functions. But I don't know if it applies to logger transforms, though. Maybe there is some interesting way to workaround this (which would be fun to find out).

[c42f]

Very interesting, thanks so much for the thoughts. It's great fun to finally have a proper design discussion about this and related issues (here, in LoggingExtras and in Base).

there can be non-filtering logger transforms

True. I think I originally named them this way in an attempt to avoid being overly vague. (I wasn't sure what a filter-or-transformer should be called). But yes we could probably do better. I wonder what the filterlogs function should be called.

Somewhat fun fact is, like transducers (which are "reducing function transforms"), ∘ would be "flipped";

Ok this is interesting. It's a bit of a conundrum because I think the fact that this "seems surprising" and needs to be explained in transducer tutorials is a sign that users would also get confused by it here. In fact I think I originally implemented the opposite here by explicitly implementing ∘ and ComposedLogFilter and switching the order :grimacing:. The "logger transform" idea cleans up the implementation a lot.

[tkf]

It's a bit of a conundrum because I think the fact that this "seems surprising" and needs to be explained in transducer tutorials is a sign that users would also get confused by it here.

It was interesting to realize sink-oriented-ness of the loggers is very close the push-based approach of transducers. I guess reducing function is kind of a sink after all.

I agree that source-oriented API is more intuitive to use. My current thinking is that there is nothing wrong about source-oriented interface (e.g., iterator) as a high-level/surface API. You can always convert sink/source-oriented APIs back and forth as long as there is "adjoint" relationship (this realization led me to this PR https://github.com/JuliaLang/julia/pull/33526 that suggests to convert iterator transforms to transducers just before foldl).

explicitly implementing ∘ and ComposedLogFilter and switching the order

Maybe flipping the order is OK, as long as you don't provide the call overload. I think this would mean to expose filters as "message source transforms" which are the adjoint of "logger (= message sink) transforms." Intuitively, it's just something like vector' * matrix' vs matrix * vector. But I guess whether using ∘ for non-callable types is recommended or not is not fully decided yet https://github.com/JuliaLang/julia/pull/33573.

[c42f]

You can always convert sink/source-oriented APIs back and forth as long as there is "adjoint" relationship (this realization led me to this PR JuliaLang/julia#33526 that suggests to convert iterator transforms to transducers just before foldl).

Oh, very interesting. Nice PR!