add more info to RepeatedMsgReduction

[davidelang]

trafficstars

http://www.rsyslog.com/doc/v8-devel/configuration/global/options/rsconf1_repeatedmsgreduction.html

I think this defaults to off, but the page doesn't make that clear.

This is really something that breaks a lot of log processing and so should be discouraged

first, it would help to have a sample showing several logs, then the same thing with this turned on. One thing is that I believe the behaviour changed over versions. In very old versions the message logged was just "last message repeated N times", but I believe around version 4 or so it changed to include the message after this so you could tell what message was repeated.

then there should be a blurb saying that while turning this on can save some space in logs, most log analysis tools need to see the repeated messages, they can't handle the "last message repeated" format. This is a feature that worked decades ago when logs were small and reviewed by a human, it fails badly on high volume logs processed by tools.

[Whissi]

Real life problem example: http://www.fail2ban.org/wiki/index.php/Fail2ban:Community_Portal#Count_.22Last_message_repeated_N_times.22_correctly

[davidelang]

The solution to that is to avoid the use of RepeatedMsgReduction. What you are seeing there is an attempt to work around the problems that it causes.

Unfortunantly, when you start getting logs delivered from multiple machines (so that they are interleved), or out of order (very possible with batch processing of logs) determining what the 'last message' was in order to treat 'message repeated X times' as X of those messages, becomes hard or impossible.

This is why this now defaults to off in rsyslog, you just avoid this entire problem

In addition, if you do turn it on, rsyslog appends the line to the 'message repeated X times' message so that you actually can tell what the (beginning of the) message was.

But the easiest thing to do is pretend that this monstrosity of a standard never existed :-)

David Lang

On Mon, 5 May 2014, Thomas D. wrote:

Date: Mon, 05 May 2014 17:10:42 -0700 From: Thomas D. [email protected] Reply-To: rsyslog/rsyslog-doc <reply+i-27875432-f09b95c90f0673a0ff0de94208a53b30abd0825e-2208335@reply.g ithub.com> To: rsyslog/rsyslog-doc [email protected] Cc: davidelang [email protected] Subject: Re: [rsyslog-doc] add more info to RepeatedMsgReduction (#19)

Real life problem example: http://www.fail2ban.org/wiki/index.php/Fail2ban:Community_Portal#Count_.22Last_message_repeated_N_times.22_correctly

---

Reply to this email directly or view it on GitHub: https://github.com/rsyslog/rsyslog-doc/issues/19#issuecomment-42255615

[rgerhards]

I have updated the doc at least a bit, including some more of the fine details. I also took the liberty to borrow some of your text, David ;)

https://github.com/rsyslog/rsyslog-doc/blob/b8df087851a40bfc177f55726ff9a31f45b113d5/source/configuration/global/options/rsconf1_repeatedmsgreduction.rst

Not perfect, but probably better. Hopefully someone will help to go through all this legacy doc. We need to also merge this properly with the new-style. I'll carry on on all this as good as I can :-)

Rainer

On Wed, Feb 19, 2014 at 2:44 PM, davidelang [email protected]:

http://www.rsyslog.com/doc/v8-devel/configuration/global/options/rsconf1_repeatedmsgreduction.html

I think this defaults to off, but the page doesn't make that clear.

This is really something that breaks a lot of log processing and so should be discouraged

first, it would help to have a sample showing several logs, then the same thing with this turned on. One thing is that I believe the behaviour changed over versions. In very old versions the message logged was just "last message repeated N times", but I believe around version 4 or so it changed to include the message after this so you could tell what message was repeated.

then there should be a blurb saying that while turning this on can save some space in logs, most log analysis tools need to see the repeated messages, they can't handle the "last message repeated" format. This is a feature that worked decades ago when logs were small and reviewed by a human, it fails badly on high volume logs processed by tools.

— Reply to this email directly or view it on GitHubhttps://github.com/rsyslog/rsyslog-doc/issues/19 .