zio-logging
zio-logging copied to clipboard
zio
Idea: sampling of logs
Hi!
Something I often struggle with is getting the log levels in production right. Either I get way too much noise, or run the risk of not having enough context to efficiently debug production errors.
One approach to improve this that I've used in the past is to do log sampling on a per-request-basis. It more or less works by buffering all log calls during the duration of the request, but only actually emits ~1% of requests' logs if nothing goes wrong. But, if a log line with level warn >= is logged, emit everything that's been buffered. This gives you the ability to use liberal debug logging in production while still keeping your log volumes at manageable levels.
I realize that this may very well be outside of the scope for this project, but wanted to open an issue for discussion!
I will add some more ideas. We need several different loggers. One is for error happened, I need log every step information but if nothing happened, I want log nothing. Which could be done by mapError( yourLogger => saveLog...) Another one is for performance log. You log every time we call every IO or method, log running time. But don't write them out. Instead, in your logger you record average time cost and max one (with all steps log for that max one).
Ah, sorry, I am not sure if I can provide PR since I am currently working in a big company. Need to check with them to see if there's any restriction.
I did find you have a very closed implementation as the same idea as I had in your test code. :) So I shall say sorry about that I didn't find it. I have some code already to realize my idea. If my company allow will PR.
Best Regards,
William
On Wed, Nov 18, 2020 at 12:04 AM przemyslaw wierzbicki < [email protected]> wrote:
@zhiwilliam https://github.com/zhiwilliam do you want to open PR with one of those?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zio/zio-logging/issues/60#issuecomment-729418065, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXX6HZGN7EOKFU5VW7HP6DSQNIUHANCNFSM4LHE3SLA .
feel free to reach me out on discord maybe we could put something together