[develop-v3][feature] Timestamped node properties with message backlog

[mkfrey]

This issue is just a feature request and an attempt to get a general opinion about this idea.

I'm using Homie to transfer and log measurement data. Currently nodes only operate and send data when a WiFi connection is present. If WiFi connection is not present or connection is lost, the data is also lost. Also the data doesn't have any timestamp attached to it.

I am proposing for a way to allow nodes to collect data right from the start and add a timestamp to it. If there is no connection, the data and timestamp would be put into a message backlog and get send out on reconnect. The time would be aquired via an mqtt time provider. In order to allow nodes to collect data even if time has not yet been synchronized, there would be some kind of time promise that gets fulfilled once connection is established and time is synchronized.

This feature should be optional and compatible to the current code, so time synchronization should only get enabled if the feature gets used in a node.

I've already implemented something similar in a private fork, so I would be willing do the biggest parts of the implementation. Tell me what you think about it and/or how you would design the topic structure!

[mkfrey]

Here is what I imagine it could look like in practice:

homie/monitor_bath/air/$log [{"timestamp":1566400221,"values":{"humidity": "72","temperature":28}},{"timestamp":1566400181,"values":{"humidity": "73","temperature":27}}] or homie/monitor_bath/$log [{"node":"air","timestamp":1566400221,"values":{"humidity":"72", "temperature":28}},{"node":"water","timestamp":1566400181,"values":{"level": "34","temperature":32}}]

[MajorTwip]

Nice idea... The Flash won't suffer from the constant writing?

But IMHO, this should be not a part of Homie, but an additional module.

Perhaps it would be interesting to have an API.

Something like:

#include <HomieLog.h>

HomieLog temp = new HomieLog(proptemp); Where proptemp is a HomieProperty.

Then The Module would get the topic and so on from the property and would handle everything you described but not the base functionality of Homie like we know it...

[mkfrey]

@MajorTwip My plan was to store the data in RAM since it will be deleted once it was sent and I don't expect random restarts do destroy it. This should work fine at least on ESP32.

I already thought about outsourcing this into my node implementations, but it would be harder to implement, especially the time synchronization, since Homie already does all the MQTT handling.

To determine whether or not it makes sense to integrate it into Homie was the reason I opened this issue, since this capability is a bit beyond the "display the current state" approach of Homie.

[mkfrey]

From the perspective of a node my current implementation is this->addProperty("temperature", "23").addProperty("humidity", "45"); This aggregates the properties, adds a single timestamp to them and puts them into a queue which gets processed in the Homie loop. This is completely separate from the other node advertisement features since I don't need the current state in a separate MQTT topic.

[stritti]

Maybe it could be defined the size of the queue? Default could be ’0’. If it is Zero it is current handling and on the other hand the size could be limited too.

[mkfrey]

I rethought it and I think it makes more sense to implement my own class derived from Node and implement my desired features there by directly accessing the MQTT client.

Now I still need to work out how to implement the time synchronization. I think the best would be to implement and then use a callback-style API for MQTT messages sent to $implementation/custom/+/set", where the users would able to set the callback function by invoking a function like Homie.setCustomMessageHandler(messageHandler). The messageHandler callback function then would receive the value of + and the payload as parameters.

[MajorTwip]

The topic $SYS/broker/time may help you

[stritti]

Looking forward your work

[mkfrey]

@MajorTwip Is this universal? And how often do updates get published to it?

[MajorTwip]

I use VerneMQ as a broker which sends the time every Minute, and, more importantly, upon subscription.

Edit: HiveMQ But no, unfortunately it is not generally available, every Broker implements $SYS/ a bit to their liking. It's like Homie, there is an attempt to create a standard.

[kleini]

Such a feature would be very interesting for me, too. It gets even more interesting, if it can be combined with deep sleep. Waking up the device then just reads the sensor values and puts the device back to sleep immediately. Avoiding a new WiFi and MQTT connection on most wakeups will save a lot of energy. The problem then only is, that we do not have any information about the time. This will require some hardware clock consuming power again.

[mkfrey]

@kleini My approach would include time synchronization via MQTT. Measurements would include a TimePromise with millis() as reference, which get fulfilled once time is synchronized. So no external RTC would be required.

[kleini]

millis() starts at 0 after a deep sleep. So, this would not help with deep sleeping devices.

[bertmelis]

How does the time get published? Is it a simple script (cron/systemd)?

If so, you just publish to a settable node?

I feel like I'm missing a point here...

[mkfrey]

@kleini If you use a timer to wake up the device you can store the time prior to sleep and add the sleep time if you want to keep the time.

@bertmelis I wrote a small program listening for the client to publish his current time and answering with the current time if the time is more than 1 second off.

[bertmelis]

So 1 node which is settable to keep time. Another node to log messages (payload = time + message)?