Are the imrelp's MaxDataSize and global() maxMessageSize parameters related?

[deoren]

@rgerhards

In the source/rainerscript/global.rst doc we have this:

maxMessageSize

The maximum message size rsyslog can process. Default is 8K. Anything above the maximum size will be truncated.

and in source/configuration/modules/imrelp.rst we have this:

MaxDataSize <size_nbr>

Default is the global message size

Sets the max message size (in bytes) that can be received. Any messages above this size will be rejected causing the relp client to reconnect and retry.

Are those two parameters related? It looks like they are based on the "Default is the ..." remark, but I couldn't be sure (both because of the wording and because of what may be a bug).

I'd like to update the imrelp.rst doc to be a little clearer on that point and also link to the source/rainerscript/global.rst for the current default value.

What is the intent of having the MaxDataSize imrelp parameter? Presumably to offer an override to the global setting, but is it a common need to have the values to be different? I'd like to note for that parameter a common use case of differing from the global setting.

Thanks.

Related:

• https://github.com/rsyslog/rsyslog-doc/blob/master/source/rainerscript/global.rst
• rsyslog/rsyslog#1782
• rsyslog/rsyslog#1741
• rsyslog/rsyslog#2215

[rgerhards]

The doc tells what should happen, but as can be seen in the issue (and probably another one), there seems to be a bug in practice. Did not yet have time to look at it :-(

[deoren]

@rgerhards: the doc tells what should happen

So if I understand what you're saying, then the MaxDataSize imrelp parameter should default to the global() maxmessagesize parameter if you do not explicitly set it.

What is the main reason for having the MaxDataSize imrelp parameter? Presumably to offer an override to the global setting, but is it a common need to have the values to be different? I'd like to note for that parameter a common use case of differing from the global setting.

[rgerhards]

I barely remember there was a case, but essentially the sizes should be the same normally.

[rgerhards]

actually (I am at the code now), imrelp uses the global default. Maybe the issue was only with omrelp.

[deoren]

@rgerhards actually (I am at the code now), imrelp uses the global default.

Great, thanks for checking. I'll modify the wording so that the doc explicitly notes that.

Maybe the issue was only with omrelp.

Not sure. I'm guessing that outputs are not throttled based on maxMessageSize limits, just the inputs?

Based on my experience as noted on https://github.com/rsyslog/rsyslog/issues/1782#issuecomment-339124192, I had to explicitly set the imrelp MaxDataSize parameter to the value I chose for global(maxMessageSize) in order to get larger messages to flow. Relying on the imrelp module to use the default value did not appear to work.

Perhaps as part of solving rsyslog/rsyslog#1782 and rsyslog/rsyslog#1741, rsyslog/rsyslog#2215 could be implemented first and I could use that to help troubleshoot and report back additional information.

[rgerhards]

Based on my experience as noted on rsyslog/rsyslog#1782 (comment), I had to explicitly set the imrelp MaxDataSize parameter to the value I chose for global(maxMessageSize) in order to get larger messages to flow. Relying on the imrelp module to use the default value did not appear to work.

If you could re-create this as a test case (and then a debug log), I would be very interested (but beware of the backlog, I have to admit...).

[deoren]

@rgerhards: If you could re-create this as a test case (and then a debug log), I would be very interested (but beware of the backlog, I have to admit...).

Sure. I actually started work on that a while back, but other things got in the way. I'll circle back when I can. Do you want me to create a separate ticket, or just post back to rsyslog/rsyslog#1782 where I placed my other notes?

[rgerhards]

Not sure. I'm guessing that outputs are not throttled based on maxMessageSize limits, just the inputs?

overlooked this yesterday: nothing get's throttled (by design not, maybe by bug). If it is oversize, it is truncated. At least this is how it should work ;-)

[deoren]

deoren: Not sure. I'm guessing that outputs are not throttled based on maxMessageSize limits, just the inputs?

rgerhards: overlooked this yesterday: nothing get's throttled (by design not, maybe by bug). If it is oversize, it is truncated. At least this is how it should work ;-)

Sorry, I used the wrong wording. Not throttled, but truncated. This happens for inputs only, right, not outputs? This is what I meant to ask:

I'm guessing that outputs are not truncated based on maxMessageSize limits, just the inputs?

[rgerhards]

I'm guessing that outputs are not truncated based on maxMessageSize limits, just the inputs?

Essentially the answer is "yes". Maybe omfwd does it differently for historical reasons. Possibly a TODO item for review -- but one that really takes time as we need to go through all of the code base to be 100% sure. So far, all is out of my head.

[rgerhards]

no, I am wrong. Transformation can cause oversize messages. I am sure that some of the output apply a safeguard so that no oversize messages are generated.

that's the oveall rule, if that helps: if we have a max message size limit set, expect that it is enforced -- whereever.

A noteable exception is non-syslog output, like elasticsearch and databasese. There size limits make no sense as the templated text is not the actual message.

[deoren]

that's the oveall rule, if that helps: if we have a max message size limit set, expect that it is enforced -- whereever.

So to be clear, syslog outputs (omfwd, omrelp, ?) are truncated if outgoing messages exceed the threshold?

[davidelang]

In some cases yes, but not in all

you can use the exec_format() command to assign the output to a variable and then do a length() on that variable to detect this before you send.

David Lang

[deoren]

@deoren: So to be clear, syslog outputs (omfwd, omrelp, ?) are truncated if outgoing messages exceed the threshold?

@davidelang: In some cases yes, but not in all

Is that due to a bug, design or perhaps a side-effect of the transformation that @rgerhards mentioned?

[deoren]

TODO item from PR #433: Update the output module docs to mention that the global maxMessageSize parameter also applies to them as well.