Payload-dependent processing and service-specific configuration

[robdejonge]

Using a simple script that gets triggered with just an event as parameter, my UPS currently publishes this event to a fixed topic. Longer-term, I may write a more elaborate version. Trying to use this, and being fairly new to mqttwarn, I've encountered the following questions.

1. Can I trigger different actions dependent on the payload, without using custom functions? The two main events I wish to send a notification for are the powerout and mainsback payloads. I've gone through examples, documentation, etc. but it seems this is not possible. Is that correct?
2. When creating a custom functions, I've figured out how to pass key/value pairs back, but I've not figured out how (a) I can use an if mqtt-message == "mainsback" then ... else ... ; and (b) how to pass just a modified payload back.

Please pardon my ignorance. I really have tried to figure this out on my own first.

[jpmens]

It would help me to see some of the data. In order to fully understand what you’re trying to do, but without that I’ll respond this way:

1. No, I don’t think that is currently possible with mqttwarn.
2. You should be able to check for a particular value contained in the payload from within a custom function. IIRC it’s described in the section on format= in the documentation.

Judging by your mention of a “simple script”, it may actually be worth your while to move the complexity of the task into that script instead of bending backwards to get mqttwarn to do it for you.

[robdejonge]

The two main events get published as follows : topic =places/home/office/ups/events & message = powerout topic = places/home/office/ups/events & message = mainsback

Within the mqttwarn.ini file, I subscribe to the aforementioned topic. But instead of just passing on the message like that, I'd like to send out a Pushover notification with a more 'human' message such as "Power outage at home office".

In order to do that, I need a custom function that considers the value of that message and then defines the message and possibly title of the outgoing notification. But for some reason, I can't get the value of the topic and the message into the function to make such a decision. I'm sure it's something really simple, but even when I copy lines from a sample function it does not seem to be working.

The ini file lines:

[ups events]
topic = places/home/office/ups/events
targets = pushover:ups
datamap = AnotherTest()
title = {robstitle}
format = {robsmessage}

The custom function:

def AnotherTest(topic, message): 
    specifier = "it didn't work" 
    if type(topic) == str: 
        specifier = "matched condition"
    if type(message) == str: 
        specifier = "matched condition"

    hetbericht = "we received: " + specifier
    return dict(robstitle="This is Rob's topic", robsmessage=hetbericht)

In the above, the dictionary passed back does work. But neither the topic nor the message seem to be passing through, so the if statement does not work.

What I would like to do is something simple like this:

if message == "powerout": 
    hetbericht = "Power outage at home office. " + minleft + " minutes left at current load." 
elif message == "mainsback": 
    hetbericht = "Mains power restored. Power was down for " + minout + " minutes."
else: 
    hetbericht = "Received the following unrecognized message from the UPS: " + message 

(where values for minleft and minout would be defined within my function, of course)

But this all depends on somehow being able to 'see' the topic and message within my custom function.

Again, pardon my ignorance.

[jpmens]

I think what you want is along these lines:

functions = 'rob.py'

[ups events]
topic = ups/events
targets = log:info
datamap = rob_data()
title = {robstitle}
format = rob_convert()

rob.py

def rob_data(topic, srv):
    return dict(robstitle="Hola Rob")

# format = 
def rob_convert(data):
    message = data['payload']

    if message == 'hello':
        s = 'Hi there!'
    else:
        s = 'Sorry to see you go'

    return s

[robdejonge]

Thanks for your help!

Let me try to understand this.

1. rob_data() is called from the datamap line, because that’s when a function will get access to the topic information.
2. Would a function called from the format line not have access to this information through the data item, the way you access the payload there too?
3. Can you clarify what the srv parameter is, in the definition of the rob_data() function?
4. A function called by the datamap line should always return a dictionary, and the keys can be used in setting the title and format lines?
5. Is the aforementioned dictionary accessible in a function called from the format line? If so, how?
6. Is the data item also accessible in a function called from the datamap line? If so, do I pass it as a message parameter or otherwise?
7. If not obvious from the answers to the above, could you share why I wouldn’t define everything in the function called by datamap and just pass key/value pairs for the title and format definitions?

Sorry for all the questions.

Apologies for lack of markup in this response. Typing this on a mobile phone. Will clean up tomorrow when at my desk.

[jpmens]

I'll try and give some answers, but I may be wrong on an account or two ...

1. Correct
2. Indeed, and the data it gets is like

{'_dtepoch': 1535911436, '_dtiso': '2018-09-02T18:03:56.867757Z', 'topic': 'ups/events', '_dthhmm': '20:03', 'robstitle': 'Hola Rob', '_ltiso': '2018-09-02T20:03:56.868138', 'payload': u'hell', '_dthhmmss': '20:03:56'}

3. The srv parameter should be described in the README; it's an object with some helper functions. So, you can, say, srv.mqttc.publish("rob", "hello", qos=0, retain=False) publish a value to the MQTT broker mqttwarn's connected to.
4. Correct
5. I don't think so. You can format = {rob_text} but that value must exist already, possibly from the datamap
6. No that's not possible I think, and I believe you've disclosed a problem: it ought to be. Do me a favor please: if you cannot solve your issue with this hints, please add a feature request that we add the message payload to the datamap input.
7. I think number 6 just disclosed that.

Your questions do not bother me in the slightest; the only thing is I've probably forgotten far too much about mqttwarn meanwhile, and it's apparent that we have to work at the documentation.

If you feel like adding bits, I will gladly welcome a pull request for better documentation.

[jpmens]

The function you're looking for is alldata as a replacement for datamap:

[ups events]
topic = ups/events
targets = log:info
alldata = rob_alldata()
title = {robstitle}
format = {robstitle} -> {robsUPS} is up at {_dtiso}

rob.py

def rob_alldata(topic, data, srv):
    print("TOPIC", topic)
    print("DATA", data)
    return dict(robstitle="Hola Rob", robsUPS="ups #3")

output

2018-09-02 20:18:03,740 DEBUG [mqttwarn] Section [ups events] matches message on ups/events. Processing...
('TOPIC', 'ups/events')
('DATA', {'_dtepoch': 1535912283, '_dtiso': '2018-09-02T18:18:03.740952Z', 'topic': 'ups/events', '_dthhmm': '20:18', '_ltiso': '2018-09-02T20:18:03.741298', 'payload': u'hello', '_dthhmmss': '20:18:03'})
2018-09-02 20:18:03,742 DEBUG [mqttwarn] Message on ups/events going to log:info
2018-09-02 20:18:03,744 INFO  [log] Hola Rob -> ups #3 is up at 2018-09-02T18:18:03.740952Z

[robdejonge]

I think I'm starting to understand this a little better now. I'd be happy to try and put together some documentation on custom functions. I spent some time this morning trying to understand things, let me attempt to summarize below.

A topic section in the INI file can have properties set as per the table at the bottom of this section. The targets, topic and qos properties can not be defined with a function.

Topic-section properties that can call a custom function

• datamap : dictionary, or a function that returns a dictionary
• alldata : dictionary, or a function that returns a dictionary
• filter : boolean, or a function that returns a boolean
• title : string, or a function that returns a string
• format : string, or a function that returns a string
• priority : see below
• image : see below

Data mapping functions

Both the datamap and the alldata properties in a topic section can call a function which returns a dictionary. The keys in this dictionary can be used when describing the outbound title and format properties of the same topic section.

A function called using the datamap property, gets access to the topic string and the service object. Functions called using the alldata property, get access to the data dictionary in addition to that. It's not clear to me why both of these exist, as alldata seems to be the more extensive environment to use.

• topic: again, seems superfluous, as data['topic'] contains the same value
• data: provides access to some information of the inbound MQTT transmission, more detail here
• service: could not find any documentation, but reading the code this provides access to the instance of the paho.mqtt.client.Client object (which provides a plethora of properties and methods), to the mqttwarn logging setup, to the Python globals() method and all that entails, and to the name of the script.

Filter functions

I believe a function called from the filter property in a topic section needs to return False to stop the outbound notification. It has access to the topic and the message strings of the inbound MQTT transmission.

Output functions

Both the title and the format properties in the topic section can contain a string where {bracketed} references get resolved using the dictionary returned from a data mapping function. Or they can call a function that returns a string that may or may not contain such references. The functions called here do not have access to the actual dictionary returned from data mapping functions though.

The priority and image properties in the topic section, I've not been able to better understand. I am not sure what these functions get passed, and I think the output is highly dependent on the target that is using the information.

---

Further following your request to raise a feature request : As mentioned above, it is unclear to me why the datamap property exists, given alldata provides a more extensive environment so I don't see why I wouldn't use that. Please clarify if I'm missing something. Happy to raise the feature request, but it would just bring the environment with which the function is called on par to alldata.

Thoughts on implementations : Trying to think how to best implement custom functions, I've come up with the following scenarios :

• If you can provide the required service output (title, format, priority and image) based only on the MQTT topic and payload (message), build a custom output function.
• If this is not enough, build as much of your logic in a data mapping function. A function called by the alldata property has access to a bunch of stuff that makes this the ideal place to put all your logic. Add your loading of additional information sources here too, for example when a UPS fails you could trigger a data mapping function that subsequently consults the UPS for the number of minutes left. An alternative to this would be to use JSON in the output of the UPS, but this is not always an option.
• To format service output, you can either have your data mapping function return a dictionary containing key/value pairs for the title and format strings, or (arguably more elegantly) build your dictionary with a data mapping function and use an output function for conditionals based on topic and payload only.

[jpmens]

alldata exists because when we noticed datamap didn't suffice we wanted to ensure backward compatibility, so didn't remove it.

[robdejonge]

@jpmens Your thoughts on the write-up above for adding into documentation as a first release of a section titled "Custom functions" ?

[jpmens]

Apologies: I meant to comment on that but it slipped.

Are you willing to file a PR with lots of copy/paste? :-) You've described it very well.

[robdejonge]

I'll have to figure out how to do that, but sure!

[jpmens]

The easiest way for you, in this case, is to hit the "edit" (pencil) button directly on the README.md in the repository itself. Github then allows you to edit and you can finalize the edit by creating a pull-request.

[robdejonge]

First draft at https://github.com/jpmens/mqttwarn/pull/331, comments most welcome. I want to make more edits before this gets pulled in. (wait, doesn't that mean I shouldn't have created a PR yet? sorry, i'm new to this!)

[robdejonge]

Questions about mqttwarn itself, the answers to which I'll incorporate in my PR once clear to me.

---

Above, I wrote :

• datamap : dictionary, or a function that returns a dictionary
• alldata : dictionary, or a function that returns a dictionary
• filter : boolean, or a function that returns a boolean
• title : string, or a function that returns a string
• format : string, or a function that returns a string

I just assumed that since title and format accept a string OR a function, this was also the case for the other 3 properties.

Is that true, can I for example add do this in an INI file :

[section-name]
topic = some/topic
datamap = { "key1" : "value1" }

Or can it only be a function?

---

Above, I wrote :

The priority and image properties in the topic section, I've not been able to better understand. I am not sure what these functions get passed, and I think the output is highly dependent on the target that is using the information.

Can someone add some insight into these for me?

[rgitzel]

Hey @robdejonge nice work! I started write up some PR comments this morning, will work on it a bit more.

I was definitely confused by "dictionary, or a function that returns a dictionary" What do you mean by that? By my understanding, those properties are just names of functions.

OH!! I didn't know about title possibly being a function. I wonder when that is used.

But no, properties like alldata are function names only. If you ever wanted to 'hard-code' a value, as in your example, I guess you would write a function that returns a constant.

(It would be really hard, I suspect, to allow Python code in the .ini file, which is what you'd need to do to allow dictionaries to be specified there.)

[jpmens]

@robdejonge I think this table answers most of your questions?

Option	M/O	Description
targets	M	service targets for this SUB
topic	O	topic to subscribe to (overrides section name)
filter	O	function name to suppress this msg
datamap	O	function name parse topic name to dict
alldata	O	function to merge topic, and payload with more
format	O	function or string format for output
priority	O	used by certain targets (see below). May be func()
title	O	used by certain targets (see below). May be func()

datamap = { "key1" : "value1" }

I don't think that will work as we expect a function name.

The priority and image properties in the topic section, I've not been able to better understand. I am not sure what these functions get passed, and I think the output is highly dependent on the target that is using the information.

That is quite accurate. Both values are used only in very few services, and I think we should not explain them "gobally". (Actually we should probably at time point redesign services to be able to obtain a "service-specific" configuration without having to pollute the configuration with settings used only once or twice.