omhiredis mode=template doesn't work as expected

[mostolog]

Hi

If I'm not wrong, each action could use template property to set how messages will be written on output. For example:

template(name="redis" type="list") {
    constant(value="app=")
    property(name="syslogtag")
    ...
}
action(
        type="omhiredis"
        mode="queue"
        template="redis"
        server="myserver"
        serverport="6379"
        serverpassword="password"
        key="my-redis-list"
)

According to http://www.rsyslog.com/doc/master/configuration/modules/omhiredis.html (and perhaps on other modules too, like omkafka), template can also be used (with mode="template") to define which command is sent to Redis, for example:

template(name=”redis” type=”string” string=”LPUSH %programname% message”)

Sadly, seems the template must be a string and doesn't work properly with spaces.

Does it makes sense template to behave different according mode configuration? It won't be more consistent to have a command property or something else (which just receives the message formatted by output)?

Regards

[rgerhards]

Unfortunately, I do not really understand what you mean, especially starting from "doesn't work properly with spaces" part of the issue.

Can you please elaborate, best with an example of what you mean. Also, keep in mind that if this is redis specific, you possibly need to provide some background in order for non-redis folks to understand.

[mostolog]

Quoting http://www.rsyslog.com/doc/master/configuration/modules/omhiredis.html

Note this mode has problems with strings with spaces in them - full message won’t work correctly.

Example: https://github.com/rsyslog/rsyslog/issues/1201#issuecomment-255349193

[rgerhards]

OK, we need to straighten some things here. This is not a problem with templates per se, but rather an issue with a specific functionality in omhiredis or redis itself. Templates are a very generic vehicle, and it is intentional to not make them specific to any plugin.

The real question is IMHO what specific functionality redis requires to make this work, e. g. what would you like to tell the template processor (or more precisely the property replacer) to solve the issue redis has. For example, do you need an option

• to remove spaces that are present in the property
• to escape spaces and, if so, how
• replace spaces by some other char?

Once we know that, we can consider adding a feature request that this gets implemented in the generic property replacer code (if not already present).

@bobthemighty, @mostolog any idea?

[rgerhards]

@taotetek sorry, I forgot you in my previous post. Would be great to have your feedback.

[mostolog]

Actually I just tried LPUSH key this is a message on Redis and it doesn't work as I expected (it inserts multiple values while I was expecting just one message), but LPUSH key "this is a message" does. Going to test template with quotes right now.

Anyway...why template must be of string kind? (it's just a omhiredis issue?) Sorry for misunderstanding!

[rgerhards]

@mostolog To me it is important that this is an issue very specific to omhiredis; it does not refer to anything with the template processor.

The output of templates are string, because everything in syslog is a string. Traditionally, there is no binary data in logging (usually even dumps are base64 encoded). Do you think omhiredis needs other data types, e.g. integer?

[mostolog]

I just mean It didn't work when I used template type="list", but considering how well I tested, don't trust me until I got news...(working on it ATM)

[rgerhards]

@mostolog ah, you didn't notice that. It really doesn't matter to the outcome if you use type="string" or type="list" in the template definition. This is just how you define the internal structure. All of these types are actually compiled into an identical structure. In the light of this, I'd say that your templates are different, and they are obviously in the sample you gave above (one starts wtith "app=", to other one with "LPUSH").

[mostolog]

Update: This is something I expected to work, but it's failing

template(name="redis" type="string" string="LPUSH %syslogtag% \"this is a message\"")

127.0.0.1:6379> LPOP mytag
"message\"

template(name="redis" type="string" string="LPUSH %syslogtag% 'this is a message'")

127.0.0.1:6379> LPOP mytag2
"message'"

Once this works (or someone explains why not), I will try the mode="list".

[rgerhards]

@mostolog the whole thing looks weired, because

• there are \n where none are inside the template
• the message is truncated

I don't know how you obtained the result. I would suggest to add

action(type="omfile" file="/var/log/template-test" template="redis")

to rsyslog.conf and see what actually gets written to that file. I would bet more than a single beer that this is not what you showed above ;-)

[mostolog]

Hi

I'm not sure if you owe me a beer, but that's what I just tested:

template(name="redis2" type="string" string="LPUSH %syslogtag% this does not work")

ruleset(name="json2redis"){
    action(type="omfile" file="/test" template="redis2")
    action(
        type="omhiredis"
        mode="template"
        template="redis2"
        server="server"
        serverport="6379"
        serverpassword="password"
        key="mykey"
        action.resumeRetryCount="-1"
        action.reportSuspension="on"
        queue.maxdiskspace="1G"
        queue.type="LinkedList"
        queue.filename="omhiredis.qi"
        queue.SaveOnShutdown="on"
    )
}


rsyslog$ tail /test
LPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not work...

redis$ LPOP tag
"work"

I doesn't work neither when setting template string="LPUSH %syslogtag% "this does not work""

[rgerhards]

Well, all I can see is that the rsyslog core generates the template correctly. So there is no error in template generation.

The question you actually seem to be after is what you need to send to redis so that it does what you want it to do. This I do not know. If, however, you can tell me what you would like to send, I can advise on what you need to put into the template.

But you need someone with redis know how if you do not know what to send. That's not really a rsyslog issue.

Rainer

Sent from phone, thus brief.

Am 31.10.2016 17:21 schrieb "mostolog" [email protected]:

Hi

I'm not sure if you owe me a beer, but that's what I just tested:

template(name="redis2" type="string" string="LPUSH %syslogtag% this does not work")

ruleset(name="json2redis"){ action(type="omfile" file="/test" template="redis2") action( type="omhiredis" mode="template" template="redis2" server="server" serverport="6379" serverpassword="password" key="mykey" action.resumeRetryCount="-1" action.reportSuspension="on" queue.maxdiskspace="1G" queue.type="LinkedList" queue.filename="omhiredis.qi" queue.SaveOnShutdown="on" ) }

rsyslog$ tail /test LPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not workLPUSH tag this does not work...

redis$ LPOP tag "work"

I doesn't work neither when setting template string="LPUSH %syslogtag% "this does not work""

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rsyslog/rsyslog-doc/issues/271#issuecomment-257341255, or mute the thread https://github.com/notifications/unsubscribe-auth/ABadi-Aa-ojpusIfhYobeKF4Cke7P5n2ks5q5hV5gaJpZM4KefrN .

[mostolog]

Using redis-cli, I think this is self-explanatory:

127.0.0.1:6379> LPUSH key this is a test
(integer) 4
127.0.0.1:6379> LINDEX key 0
"test"
127.0.0.1:6379> LPUSH key "this is a test"
(integer) 5
127.0.0.1:6379> LINDEX key 0
"this is a test"

[rgerhards]

@mostolog no, this is not self-explanatory to me, because the template generates the same string you use in your manual sample. So we need someone with redis know-how to understand why the same string does not work.

The only thing I may come up with is the question if a \n is needed at the end of the string? That is not part of your template...

[mostolog]

LOL. I tried with \n and didn't work neither.

I'll try to be more clear this time. If template is defined as:

template(name="redis2" type="string" string="LPUSH "%syslogtag% %msg%"") (notice quotes. without them, it's working as it should)

and msg contains spaces. eg:

2016-11-02 10:00:00 host tag rgerhards should like this message

I expect it to add msg as a single item:

LINDEX key 0 should return "2016-11-02 10:00:00 host tag rgerhards should like this message"

instead of

"message"

Using a template without quotes, then it should add multiple items (splitted by " "), as it actually does.

[rgerhards]

Well, I think we misunderstand here. My point is that this is in no way related to the template engine and possibly also not to omhiredis. So you need help from someone with redis knowhow or someone who knows and/or contributed to omhiredis.

[mostolog]

I'll copy @bobthemighty @taotetek to get their feedback. I still think it is almost an omhiredis issue.