Logging Damper: avoid heavy logging when appropriate

[pjanotti]

trafficstars

The service needs to provide good observability to its users but it needs to be careful about not overwhelming the system with logging/metrics/traces. In this regard logging is typically the most problematic one. Ideally we should provide a global solution instead of doing piece meal (the typical log every x times or x seconds).

[tigrannajaryan]

I found that this approach works well for high-perf services that need to log:

1. Have 2 modes of operation for each log statement, regular and throttled.
2. In regular mode all logs are printed.
3. In throttled mode you only print one exemplar followed by a counter of "similar" logs since last print.
4. Frequency of logs triggers movement from regular to throttled mode and back. Frequency is counted per source code line where log statement is located.