Proposal: Read initial state of properties from retained value of MQTT broker

[euphi]

I think it would be great if Homie could read the initial value of a property from its retained MQTT message.

Example use case: If a value can be set from a homie device (but is not a sensor value), the homie-device should restore it after restart. (E.g. a set-temperature for a thermostat.)

So:

1. homie-esp8266 publishes a property as retained message. E.g. homie/Labor/Thermostat/SetTemp 22.10.

2. homie-esp8266 is restarted (for whatever reason). The MQTT broker and the automation system (e.g. openhab) continue to run. (Note: Thanks to retained messages, the case for restart of the automation system is already covered.)

3. After restart, homie-esp8266 subscribes for its property messages (at this step, it does not yet subscribe for the ../set message!)

4. So, if a retained message is available, it is received by homie-esp8266 and is used to set the initial value of the property.

What do you think about the proposal?

Open questions for an implementation:

• Should homie-esp8266 automatically subscribe or should there be a getRetainedValue() method?
• Should homie-esp8266 use the handleInput(...) method to handle the received retained value or should there be an extra callback, e.g. handleRetainedValue(...)?
• When is the correct time to unsubscribe again? Inside the handleRetainedValue(...) method?

Other approaches to find a good solution for the use case "persist values in homie":

• Store locally in SPIFFS or EEPROM --> drawback: Not useful for data that is often changed (Use lazy write?)

• Make the home automation system send the persisted values in a set-message after a reconnect from a devicve. --> drawback: Quite complicated to configure.

• Variant: Let the home automation system always send a "set" message if a value has been changed by the home-device itself. --> drawback: homie is supposed to "confirm" a set-value by sending the set value as mqtt message. So this would result in a loop.

[bertmelis]

It would be very welcome for me too. I'm thinking of my gasmeter pulse counter: the total count (= representing the current meter total) is reset when restarting the ESP. Hence I have to manually set the total value again. And as my logging is working with diff values, there's always a spike when doing so, which I have to "delete" too. This could be avoided by resuming using the retained value.

My preferred method would be something analog as the .settable implementation. I think it's up to the integrator to decide whether is should be implemented or not.

Homienode.advertise("property").settable(SetProperty).setRetainedValue(RestoreProperty);

bool RestoreProperty(const HomieRange& range, const String& value) {
  //handle restore
}

[marvinroger]

Even though Homie for ESP8266 is an high level abstraction, I'm not sure everything should be abstracted. Indeed, in your case, the data is specific to the device, so it makes no sense to store it in the broker, knowing this is very simple to store data on the ESP8266 (SPIFFS, EEPROM, see https://github.com/marvinroger/homie-esp8266/issues/154)

[euphi]

As my thermostat is now running fine, I still think the most elegant solution to restore the last value of the set-temperature is to read the retained value from MQTT.

I think, this should by added to the homie convention as recommended way to restore data that is set to the device.

Reasoning:

• data that is sent from the automation system to the Homie device belongs to the automation system.
• the MQTT broker is the proxy of the automation system.
• if a Homie device restarts, it needs to know the last value of settable properties.
• the "pessimistic feedback" approach of Homie has the effect that the valid value is always sent from Homie to the broker.
• if this value has been sent retained, it can easily be read from MQTT during startup.

[luebbe]

An additional plus is that it wouldn't require a reboot to change "trivial" config values. Currently I have a display node that fetches values from a MQTT topic and displays them. Right now the topic is stored in the config in order for the display to start up and show the values automatically. Whenever I want to change the topic, I have to send a new config and reboot the device. If the topic can be read from a retained "set" property, it would allow the device to start in a defined state plus the topic can be changed easily without a reboot.

[timpur]

In theory, homie should receive the retained message from any /set property? if this is not the case it might be a bug and ill look into fixing this?

[jamesmyatt]

I think that the MQTT broker can only send a retained message again (or any other sucessfully completed message) if the device re-connects with the "clean start/session" flag set, i.e. if the broker thinks the message has never been sent before. Alternatively, you could unsubscribe then subscribe again to that topic.

[luebbe]

@Nzbuu That would explain why mqttfx doesn't display some messages when I just subscribe to a particular subtopic after having caught "all" (retained) messages with a homie//# subscription before. It always worked with mosquitto_sub, but since mosquitto_sub re-connects every time, it's now clear to me, why it worked. Thanks, learned something again. This was driving me mad.

[timpur]

I think this is a bug now, just noticing that it doesn't pickup on retained messages in the nodes, i recall that it use to. Will fix.

[timpur]

Actually i think i misunderstood whats going on here. @euphi what did you want from homie?

[euphi]

In my opinion it would be the best solution that:

• Homie shall have a flag that a property is reloaded from retained value.
• If this flag is set and Homie reconnects to the broker (or only on first connect after reboot?), Homie shall read the property value from its retained value on MQTT.

Best way to set the flag would be to add this to the PropertyInterface.

The feature is most useful on settable properties, but it can also be useful for properties that are not settable over MQTT, but settable in other means, e.g. user-interaction.

[jamesmyatt]

It turns out that it is possible to obtain the retain message by re-subscribing without unsubscribing first (according to the v3.1.1 spec)

If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3].

I assume that Homie re-subscribes to everything every time it connects, since it won't know what it's currently subscribed to. Hence, you should get a retained message at that point.

However, then there's always the issue that the broker may discard the retained message at some point, which it is permitted to do by the specification when it was published at QoS 0:

[The Server] SHOULD store the new QoS 0 message as the new retained message for that topic, but MAY choose to discard it at any time - if this happens there will be no retained message for that topic [MQTT-3.3.1-7]

[euphi]

Ok, as the homie-device shall reread its own values, its up to the device to set retained flag and the QoS level.

[timpur]

So just add a flag to show when the message is the loaded retained messages for the set handler ?

[euphi]

No. An additional "one-time" subscription for the property-topic without set.

I use my thermostat for an example:

homie/wz_thermo/Thermostat/$properties SetTemp:settable,Mode:settable

Both Properties of the "ThermostatNode" has been set by my controller (openHAB) - and then confirmed by Homie with a sustained message:

homie/wz_thermo/Thermostat/SetTemp 20.90
homie/wz_thermo/Thermostat/Mode Auto_OFF

When there is a restart, the device needs to know the current SetTemp and mode to display it correctly.

The simplest way would be to reload the value from these messages.

Why not homie/wz_thermo/Thermostat/SetTemp/set?

• There is no need to send this message sustained, because it is just a command that is invalid after confirmation.
• The SetTemp can also be changed by the Thermostat itself, then no set topic has been sent.

[timpur]

I see now. Let me look into this and I'll get back to you about the best solution (in my opinion).

[jamesmyatt]

I think that seems OK for this relatively simple situation, but it's worth considering that there might be other ways to change the properties. For example, you might also have buttons on your thermostat to change the temperature directly. How would it work in this situation?

I see the "set" messages as commands. They should be executed once and then discarded.

[nerdfirefighter]

@Nzbuu I tend to agree with you. the examples provided, the gas and thermostat both in my mind would be best done as Custom Settings (stored on the device) for the following reasons:

1. they don't change that often. (really how often do you change the themo set temp of the heater/cooler)
2. They both (should) have a requirement to work even when the MQTT is not available e.g. what happens if the gas monitor is rebooted while the network is down, you miss pulses because when it reconnects and get the last posted values, which is now a number of pules behind. or you have to write more code to handle this gracefully. for Thermostat, your example of if there is local control as well as remote control, the source of truth has to be the device its self, to the message it posts.

MQTT persistence is really more for the consumer so they can get an initial value between the time they start consuming the MQTT channel and when the device sends a new message.

Any way, my two cents on the matter and sorry for playing devils advocate.

[ingoogni]

Agree with @Nzbuu the state of things may have changed during the down period and or reset and you'd be working with stale data. Also a "drop out" should be seen as a "drop out" and not be smoothed away.

In analogy with @bertmelis gas meter, what I do with a water flow. Every second I push the flow and measurement period to the database. Then I use Postgresql's listen/notify pub/sub system to publish the total volume back tot the device that measures and that displays it. Both messages contain an uuid, when they don't match something went wrong in a measurement series. (currently not via MQTT and Homie-ESP but direct from ESP to DB and back)

[jamesmyatt]

These are all really good examples of cases where you might want to retrieve a property value that isn't updated via a "set" command at all. It would be great to be able to restore the last value at power-up without storing every increment in SPIFFS. I see this as storing a "state" variable (or dictionary) rather than a "command" variable.

I think that it's just tough-luck if you miss some pulses while the device is powered-off. If that's a problem, then you need some kind of battery back up.

[euphi]

The feature to reread values from sustained message is meant for a restart of the Homie device - not a restart of the MQTT broker.

In detail:

@Nzbuu Thats exactly the reason why I think the property message from homie to controller shall be re-read and not the set message.

@nerdfirefighter The feature to reread sustained values is meant for failure and restart of the Homie device. If the MQTT broker is unavailable there is no need to read values after the reconnection to the broker, because the values are still stored in the homie device.

@ingoogni Most data can't change during the down time of the Homie device. If it is a settable property the homie device must confirm the value to make it valid. This is not possible as long as the homie device is unavailable. If it is a value that is "created" at the Homie device, it can't be updated when the Homie device is down. Then, in your gas/water meter examples you loose "counts", however solving this with redundancy etc. is beyond the scope of Homie.

The only "tricky" situation I see is if both MQTT broker and Homie device are restarted at the same time (e.g. due to power fail) AND the MQTT broker takes signifcant longer to restart than the Homie device, so the Homie device can "count" several ticks before it can read it old value from MQTT. However, also in this case, rereading the sustained value helps solving the problem: After MQTT reconnction, the homie device can add the counted ticks during to the retained value and update the value acconrdingly.

[nerdfirefighter]

On 11 Jan. 2018 23:57, "James Myatt" [email protected] wrote:

These are all really good examples of cases where you might want to retrieve a property value that isn't updated via a "set" command at all. It would be great to be able to restore the last value at power-up without storing every increment in SPIFFS. I see this as storing a "state" variable (or dictionary) rather than a "command" variable.

Yet to see any examples of why you would do this. It seems to me like its bloating a really nice lite weight code base.

I think that it's just tough-luck if you miss some pulses while the device is powered-off. If that's a problem, then you need some kind of battery back up.

I think you have missed the point here. Point of the example was, most of the time the device where you need to start from a last know good value. Is likely a device that needs to operate with out relying on mqtt to configure its start up values.

E.g. i have a relay with its own physical button. I have a requirement that it returns to its last state on power up. I keep the source of truth for state in a custom config item. That way if there are any mqtt issues, or the device boots faster on recovery then my server, wifi, etc it will return the relay to its saved last state. It also ensures that if the physical button is used while mqtt is down, the last good state is still tracked.

[ingoogni]

The water does not stop flowing when power fails. On restart the count continues but I have 100l more in the tank than the display tells because it continued with old data. Not Homie or MQTT problem to solve IMO.

[nerdfirefighter]

On 12 Jan. 2018 00:04, "Ian Hubbertz" [email protected] wrote:

The feature to reread values from sustained message is meant for a restart of the Homie device - not a restart of the MQTT broker.

In detail:

@Nzbuu https://github.com/nzbuu Thats exactly the reason why I think the property message from homie to controller shall be re-read and not the set message.

@nerdfirefighter https://github.com/nerdfirefighter The feature to reread sustained values is meant for failure and restart of the Homie device. If the MQTT broker is unavailable there is no need to read values after the reconnection to the broker, because the values are still stored in the homie device.

@ingoogni https://github.com/ingoogni Most data can't change during the down time of the Homie device. If it is a settable property the homie device must confirm the value to make it valid. This is not possible as long as the homie device is unavailable. If it is a value that is "created" at the Homie device, it can't be updated when the Homie device is down. Then, in your gas/water meter examples you loose "counts", however solving this with redundancy etc. is beyond the scope of Homie.

The only "tricky" situation I see is if both MQTT broker and Homie device are restarted at the same time (e.g. due to power fail) AND the MQTT broker takes signifcant longer to restart than the Homie device, so the Homie device can "count" several ticks before it can read it old value from MQTT.

Thats my point.

However, also in this case, rereading the sustained value helps solving the problem: After MQTT reconnction, the homie device can add the counted ticks during to the retained value and update the value acconrdingly.

This adds complexity because you have to write code for that logic and related data validations. And also work out when to do that logic. By using a custom setting to store the source of truth you remove the need for this over conplication of logic.

[nerdfirefighter]

On 12 Jan. 2018 00:16, "ingoogni" [email protected] wrote:

The water does not stop flowing when power fails. On restart the count continues but I have 100l more in the tank than the display tells because it continued with old data.

I get your point but that is a bad example as that is the measure of the capacity of the tank which you cant reliably get from flow rate in to the tank due to this situation indicated. You would measure the flow to calculate how long it would take to fill/empty the tank and measure the capacity of the tank to know how much you have in the tank.

Regardless of if the value comes from the mqtt last set or on board its still going to be wrong value in your example. You would need to send a new set command to the device to advance/correct it to resolve the issue. Which is different to reading back the last state.

[ingoogni]

Yes. The only way to know a state is to measure/determine it and not depend on data from the past, regardless where they come from DB, Broker, EEPROM...

[timpur]

I might be misunderstanding but the current state of the property should be the same as the /set.

The property should be in sync with the /set. The property is there to show feed back of a changed state from /set (for settable properties). If you have a property that is settable, then you should always use the \set to set the a new state internal or external. This way you always follow the same flow to change states and you can get the latest stat as the retained /set will be received and flow through the same slow, as in the set by the set handler.

Now currently there are issues with trying to follow this flow, but i believe we should be following this flow and fix the issues with homie to allow this, instead of creating a new flow. Follow the same flow for internal as you normally do from external property setting.

An issues is, if you lose connection to mqtt, pushing to /set is impossible.... Fix could be that if we are disconnected and we use /set then it will bypass to the set handler?? This way we can always use this flow if were online or offline to set new states of the device..

obvious issue is now do we send the /set state, when back online because the /set was not sent as we were offline....? I think that it will be hard to make homie 100% full proof.... We could implement a queue .... but not things are getting complex

lest find a happy medium ??

[jamesmyatt]

The problem is that there are lots of situations where it's not as straightforward as just re-executing the set command. For example, there may be no set command like in a pulse counting device.

Also, the value sent to the set command may not be the value you want to restore. For example, you may want to implement your device to handle "volume/set" <- up or down. Then there's no point in restoring "up" or "down".

[jamesmyatt]

To clarify, I think it would be very useful to be able to restore settings and last known state variables from an MQTT retained message, but I think it's going to be hard to find something that satisfies most use-cases and I don't think that just re-executing the last set commands is going to work: there are too many exceptions.

[marvinroger]

Remember that Homie is state-based. You don’t tell your device to “turn on the device”, but to put its “on” state to “true”.

Same thing for the volume.

Cordialement, Marvin ROGER

Le 12 janv. 2018 à 17:32 +0100, James Myatt [email protected], a écrit :

The problem is that there are lots of situations where it's not as straightforward as just re-executing the set command. For example, there may be no set command like in a pulse counting device. Also, the value sent to the set command may not be the value you want to restore. For example, you may want to implement your device to handle "volume/set" <- up or down. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[timpur]

I dont think homie should go there, you should be solving this through pattern not new feature.

I think ill add a flag for the set handler, so you can tell if its the restored retained message and you can decide if you want to restore this message (act on it).

You should have a volume node with properties of control (up/down) which influences the volume state property (percent), just like a dimmer has value and switch... restore the last set volume percent but not the last control (up/down) in that example, something like that.

We dont want the true current state to be store in any number of locations, on the device or on MQTT, it get complex quick....