komu.engineer
komu.engineer copied to clipboard
komuw
Discussion: Logging without losing money or context.
This is the place for comments/discussion regarding the article: Logging without losing money or context.
Please be civil.
Thanks for sharing 👍
I just want to add that this method needs a maxBytes auto-flush mechanism. Imagine the app is 3 months online without any error, but lots of logs, then it would consume lots of memory. So it would be nice to set up a goroutine to check if lineBuffer is reached to maxBytes and flush it.
Hello @EricMasonFa, thanks for your comment.
In the article, the code contains this comment;
// Note: in production, lineBuffer should use a circular buffer instead of a slice.
The idea is that you should ideally use a circular buffer[1] like armon/circbuf[2].
That way you don't run into memory issues. But your idea could also work.
- https://en.wikipedia.org/wiki/Circular_buffer
- https://pkg.go.dev/github.com/armon/circbuf
I wrote about something similar in Java https://tersesystems.com/blog/2019/07/28/triggering-diagnostic-logging-on-exception/
@wsargent Hi. Read the article and it's pretty cool. I wonder why this kind of logging strategy isn't more common.
On Thu, 28 May 2020 at 05:08, Will Sargent [email protected] wrote:
I wrote about something similar in Java https://tersesystems.com/blog/2019/07/28/triggering-diagnostic-logging-on-exception/
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/komuw/komu.engineer/issues/17#issuecomment-635049978, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHMWUODQRTBVOHMJPUXUW3RTXBTRANCNFSM4NJBL56Q .
-- Thanks,
KomuW. https://www.komu.engineer/about
Wow, this blogpost[1] is a gem. I wonder how come I had not come across it before. Thanks @wsargent for writing it up.
I especially liked this part:
The oldest reference I could find to ring buffer logging was Hugh Blair-Smith talking about the Apollo computer in Journey to the Moon:
Another function served by these auxiliary erasable memories was called the "Coroner," which recorded every instruction executed, with its inputs and results, writing over the oldest such record when it filled up. When a program "crashed," you could punch out a full record of what it was doing in most of its last second and analyze the problem at your ease. I have often wished that PCs offered such an advanced feature. - [2]
- https://tersesystems.com/blog/2019/07/28/triggering-diagnostic-logging-on-exception/
- https://authors.library.caltech.edu/5456/1/hrst.mit.edu/hrs/apollo/public/blairsmith3.htm
A different approach I've been thinking about is to store a list of log lines on a context.Context and at the end of, for example an HTTP request, check if any errors had happened, log all of the logged events in a single (structured) log statement. This would allow me to get the entire context of the logged events without having to stitch together multiple log lines in Kibana/whatnot.
The downside of the above approach is of course that a deadlocked request won't be logged. There's a possibility to add a timeout so that the statement is logged after a certain time to accommodate for that case, though.
@JensRantil That sounds like a sensible approach.
at the end, log all of the logged events in a single (structured) log statement.
This reminds me of; https://brandur.org/canonical-log-lines
Lossless Log Aggregation is the process of aggregating similar logs into a larger aggregate log.
Common metadata and values are deduplicated and merged during the aggregation.
When done effectively, this can result in a 100X reduction in volume and a 40% reduction in size.
Without dropping data.
- https://bit.kevinslin.com/p/lossless-log-aggregation
This series of blogposts by Matt Klein(of envoy proxy) are interesting and cover a similar topic to the one here;
- https://mattklein123.dev/2024/04/03/observability-cost-crisis/
- https://mattklein123.dev/2024/04/10/do-you-need-to-store-that-telemetry/