BSB-LAN
BSB-LAN copied to clipboard
fredlcore
[FEATURE REQUEST] Add UDP broadcast of log parameters
A year ago or so, we had two users who implemented a UDP transmission of parameters to use with their DIY room units to display various parameters there. I was thinking whether it would be possible to use our current MQTT functionality and just add a few command to send the same payload via UDP, probably just as a broadcast to keep it simple. The receiver(s) could then parse the payload and display whatever they need. A receiving function could also be implemented the same way so that the MQTT processing routines could be used, like for setting a parameter (DHW push after pressing a button on the room unit etc.).
Add one more user who did this just recently (https://github.com/DE-cr/BSBmonCR). :)
I'd be willing to do an implementation proposal - once some design decisions have been made:
- Shall the parameter list and send interval be shared with logging (less implementation effort) or shall a separate configuration set be implemented (my preference)?
- What data format shall be sent? The same as in logging (e.g. "364593010;01.05.2022 00:00:15;8314;Kesselrücklauftemperatur Ist;66.7;°C"), or a more concise one (e.g. BSBmonCR would use "8314:667" for the same data)?
- Or should UDP sending be more like MQTT, as fredlcore suggested? I haven't looked at MQTT, yet, but would propose to keep the UDP data as easily parseable (and concise?) as possible.
Personally, I would
a. definitely use UDP broadcasting, not sending to one (or multiple?) dedicated receivers
b. not allow parameter setting via UDP. This can already be easily done via a simple "HTTP GET /S
@fredlcore, let me know what you think.
For now, I would keep the UDP logging parameters the same as the general logging parameters. I don't think someone will be using SD-card logging and UDP logging and use different parameters on both. The issue I see here is the update interval. If you were to use UDP for updating, say, in a project like the one you mentioned, it makes a difference if you poll 20 parameters within 10 seconds (which will clog the bus and BSB-LAN) or just two or three. So I'd suggest adding a UDP update interval in the configuration. As for the data format, I think it's easiest to take the format we already use for logging on SD-card or MQTT. That way, the remote device does not need to make a look-up to find out what the parameter number means, what unit it is in etc. and transmission time should not be affected in a noticeable way.
FYI, to not clog the system, my BSBmonCR doesn't fetch+send all (of its fixed six) parameters at once every send interval, but instead rotates through them and only fetches+sends the next one in each send interval (2s for BSBmonCR). P.S. ...with such a design, the system load would be independent of the number of parameters to send.
But it would mean that certain more critical parameters would only be sent after a certain amount of time. One of the scenarios I'm thinking of is a room controller/display, maybe it's even similar to what you've been creating. If I use a ePaper display, I can put the ESP to sleep for let's say a minute or so, wake up for max. 10 seconds (if that's the UDP interval) and update the screen, then put it to sleep again. If I have to wait for the whole interval to be completed, that would take quite a bit longer, in addition to the fact that the room controller doesn't know how many parameters to expect (unless you hard-code it). Maybe if the room-controller could initiate a request of some sort, it may alleviate some of the issues, because generally I like the suggestion of the rotation...
For critical parameters, I would not rely on a UDP broadcast. For those, I would use the existing "GET /".
How is clogging the system avoided with many parameters in /D logging? (Not at all, iirc.) Maybe this isn't really a problem then, and the easiest (=least invasive) solution would be to just use the existing logging code and include some lines to also/alternatively send the very same lines that are/could be written to the log file as a UDP broadcast. Added user configuration would then be limited to UDP port number and log data destination choice (file and/or udp).
It's not critical in the sense that the actual parameter is required every x seconds, but the broadcast in itself should be done regularly and in a compact manner, so that the ESP can go to sleep again. For /D, there is no clogging prevention, only the limiting factor of 20 (IIRC) parameters means that logging would be completed within one minute. Of course if you set the interval to 10 seconds or so, it will surely get clogged.
My proposal would then be to go forward as described in my previous comment here. Let me know if I should prepare a pull request.
Which previous comment exactly and which way? The cycling variant would not provide one compact UDP transmission, but rather several ones which would make it impossible for the ESP to get all data and go to sleep again.
Use the /D logging code: add UDP sending to logfile writing, let the user configure which one (or both) to use.
Ah, ok, yes, fine! @dukess: Could you add a setting to activate logging via UDP please (similar to MQTT, no extra category, can be added to "Logging")? Thanks! For UDP port, I'd choose 6502.
https://www.speedguide.net/port.php?port=6502
Maybe it's better to make the port number user configurable. ;) This would also help to avoid collisions. We're talking about broadcasting, after all ..
Could you add a setting to activate logging via UDP please (similar to MQTT, no extra category, can be added to "Logging")? Thanks! For UDP port, I'd choose 6502.
Just add the option? Yes. sure.
@fredlcore, @DE-cr, did it: https://github.com/fredlcore/BSB-LAN/pull/457/commits/ed5ec260e29cac0d320564a06c2252a1c710c49a
@DE-cr: NetOp is a rather special IT environment application, I don't think that this will collide with our use-case. And no, as mentioned in several issues before, no more special interest configuration options that 99% of the users are never going to use, it just pollutes the configuration screen further. So either people change the port in the code if they really need to or they don't use the feature ;).
@dukess: Thanks a lot, that was really quick! But as mentioned above, please remove the port configuration options for MQTT and UDP broadcast. Thanks!
But as mentioned above, please remove the port configuration options for MQTT
Looks like i lost something important :D Done.
Maybe I just thought I had written it in the MQTT issue already, but maybe I didn't - sorry ;)! And thanks for the quick fix!
With just a single on/off value to enable/disable UDP logging: Should UPD logging disable SD card writing, or should it work in parallel? The latter would require an SD card for UDP transmissions, which doesn't make sense to me.
Should UDP logging only use datalog.txt data or also include journal.txt data?
No, UDP and SD card logging are separate of each other. But I think there's no need to differentiate because the part where it writes to SD card checks if an SD card is present. If not, it won't write. Since logging applies to the logging parameters, only datalog.txt applies. IIRC (currently away from home), journal.txt logs broadcast messages. We could send these as well, but since they do not happen at the same time as the queried logging parameters, we won't be able to send it in one go. Plus, people interested in these values can simply also query them.
So if I want to have UDP "logging" only, I would have to use this define in the config file on an ESP32 (and not have an SD card attached to it)?:
// Use SD card adapter on ESP32-based boards instead of SPIFFS flash-based storage
//#define ESP32_USE_SD
Also, an attached SD card might be used for different purposes by custom extensions. Why waste space on it when UDP-only logging is wanted?
I can understand that you don't want to clutter the /C page, but why not use a four-state setting for logging then, as I've suggested some comments ago?: logging = off / SD / UDP / SD+UDP
Yes, that would be the way to go, same as with MQTT. And that would also prevent a four-state setting because with MQTT it would already be eight states. I agree that it is a bit counter-intuitive for the ESP32 where you have to add another definement in order not to use the internal flash, but other than that, it's quite straightforward: You want logging (of whatever kind), and if you do not want to log on SD card, you simply do not insert one. As for "custom extensions" using the SD card. that's a bit far fetched. Not only because I don't know of any except maybe a handful, but what other purpose would someone use the SD card for besides logging? And even if there were such custom solutions, BSB-LAN is not going to develop into a 'Homer Car' (anymore than it already is ;) ) https://www.youtube.com/watch?v=WFZUB1eJb34
I'm not sure, @dukess, how easy it would be to implement multiple select options. If that would be possible, we could use that and make one configuration option to define the logging option as a bit pattern where users could (un-)select SD card logging, MQTT, UDP and whatever other logging options would come up in the future.
I think the actual values would be provided by the HTML code (0, 1, 2, 4, 8 etc. until 128), so storing that in a one byte value should not be an issue. Decoding it and creating the corresponding settings is maybe a bit more work, similar to VT_BIT, I guess...
Current progress: https://github.com/fredlcore/BSB-LAN/pull/457/commits/9f71bf86009fd3a71b5a36b44a7c802c8ca13f09
Right now, the checkboxes does not affect the operation of the program. We need to discuss which variables we can abandon, how to change the logic of the switches and where to place the new parameter.
Great, thanks a lot! I'm just thinking that IPWE is not really logging, so that may be a bit confusing? Did we have an extra setting for this or was it through the use of 'define'?
Yes, you're right, IPWE is not a real log. My idea was that the "logs" include all the ways to send data outside the program.
I think it's a rather special function that only few people (but including myself ;) ) are using, so it should stay in the "advanced" configuration in order not to confuse people ;)...