lxp-bridge
lxp-bridge copied to clipboard
celsworth
Add more frequent requesting of input data
Expanding from discussions in #103, it's becoming clear that it would be nice to get input data a lot more frequently than the 5 minutes the inverter sends it out automatically.
This isn't a massive job but there a few technicalities that might make this a bit tricky than at first glance.
The mobile app requests data every 10 seconds in Local Connect mode, so we'll emulate this. In order to not create a new data row on the Luxpower portal every 10 seconds, I believe only registers 0-79 (ie not 80-119) are requested by the app, so we'll do the same.
This creates a problem that the MQTT packet lxp/{datalog}/inputs/all, and database rows, and InfluxDB, are all only populated when we get those registers 80-119 (aka inputs/3).
So we'd need a way to tell all these subprocesses that data is not arriving, and if we want to fill in the various databases, we need a value to put in those columns. MQTT could just use nulls, InfluxDB can be omitted, but the database columns are NOT NULL.
I might be misinterpreting what you're asking, but the following automation worked for me :
alias: Poll Battery State 1 & 2
trigger:
- platform: time_pattern
seconds: /15
action:
- service: mqtt.publish
data:
topic: lxp/cmd/<YOUR INVERTER ID>/read/inputs/1
retain: true
- service: mqtt.publish
data:
topic: lxp/cmd/<YOUR INVERTER ID>/read/inputs/2
retain: true
mode: single
It'll send out a MQTT request to the router every 15 seconds and gets a response sent out.
This is good stuff, thanks for posting this - maybe we can get this into the wiki somewhere. This is a nice alternative to having the functionality baked into lxp-bridge.
The only downside to doing it this way and requesting only inputs/1 and inputs/2 is that the database rows and influxdb won't get updated as they trigger a new write when inputs/3 is received. That said, I think a new database row every 15 seconds would be a bit too much. InfluxDB would be okay with that though I guess.
Thanks for the tip, @Beenay25 - I'm running my own python script to subscribe to messages from lxp-bridge, and I realised that I could add an extra bit of code to periodically request data whenever I wanted. This means I can go rescue my old tablet from continued burn-in. I'm hoping this method is more reliable too, as the official Luxpower app would sometimes fall offline and stop requesting frequent updates.
This also allows me to request inputs/1 (live watts) every 10 seconds and inputs/2 (cumulative kWh) every 5 minutes, as I don't need high resolution on the latter.
I tried out the automation by @Beenay25, and while it does publish a message to the topic, it doesn't appear to save anything to InfluxDB, from which I'm building my Grafana dashboards.
Reading all 3 inputs works, but I understand it's undesirable since it also causes excessive writes to the remote Luxpower service DB.
@celsworth I wonder if it would be sensible for an MQTT command to be implemented in order to request an InfluxDB/database write? So that the user decides when they want the data to be persisted.
I wouldn't be opposed to that, that could work.
Few points to consider..
- if some of the data is stale, do we write it to Influx/database anyway or not? For example, if you're requesting inputs/1 and inputs/2 every few seconds, then requesting a db write, what do we do with the stale inputs/3 data
- what do we determine as being stale, data not updated since last write? what about if only inputs/1 has been updated? There's maybe a few edge cases to think about.
- if we want to be able to omit anything that is stale then the PR making some columns NULLable (#114) will need to perhaps be updated to make everything NULLable.. could get a bit messy
I kind of wonder if its trying to add a bit too much flexibility and making the job harder. My preferred solution is probably some sort of single routine to request inputs/1 and inputs/2 then store only that data. This could be hooked up to a new MQTT command or schedule. That's where I was trying to go with the initial PR making inputs/3 optional, so the database wouldn't cry over that missing data (which isn't very interesting to request every few seconds, and also spams Luxpower as you say).
Another vote from me on this task. I've noticed that when I click "read all" on the Chinese cloud interface, that my MQTT receives lots of updated data for sensor data too. So I'll give @Beenay25's suggestion a go and if all goes well, it'd be good if just prodding that one place will cause the rest to update more regularly :)
Ultimately I'd like to disconnect the cloud side all together. There's just something that makes me a little nervous about having so much of my house's infrastructure at the mercy of a tech guy in China right now. Geopolitics there are calm right now but all things solar etc always make me think of backup solutions should I ever need them 😅
May I ask if the automation above by @Beenay25 is still working? I've set up the same automation with my datalog ID and I'm still only seeing my entities in home assistant being updated every five minutes. Have I missed a step somewhere?
I tried running a listener on the lxp/[my ID]/inputs/1 but nothing is showing up there either :/
I've plonked a listener on lxp/[my ID]/inputs/2 and waited the full five minutes and can confirm I'm definitely receiving the automated one.
Heading home now so I'll get on the network and snoop at all the stuff flying around the broker, see if anything is obviously wrong with what I'm doing :)
It definitely still works for me!
alias: Poll Battery State 1 (W)
description: ""
trigger:
- platform: time_pattern
seconds: /15
condition: []
action:
- service: mqtt.publish
data:
topic: lxp/cmd/BA12132005/read/inputs/1
retain: true
mode: single
In the above automation you need to update <YOUR INVERTER ID>... so something like BA1234567
@celsworth Sorry for replying late — life took over :)
I totally agree regarding complexity of handling stale data.
My preferred solution is probably some sort of single routine to request inputs/1 and inputs/2 then store only that data. This could be hooked up to a new MQTT command or schedule. That's where I was trying to go with the initial PR making inputs/3 optional, so the database wouldn't cry over that missing data (which isn't very interesting to request every few seconds, and also spams Luxpower as you say).
Sounds like a good approach to me! I understand this needs #114 merged first. Is there something I could help with?
Okay, I think I've found my problem, though I don't know how to fix it.
From what I can tell, this retain: true is important. However, for some reason, everything I send, whether through a service on home assistant like above, or through a tool like MQTT Analyser on my phone, the retain: true flag is being utterly ignored. Anything with retain: true applied to it is being immediately changed to retain: false somewhere. I'm at a bit of a loss for how to fix this but I assume it's some configuration mistake with my mosquito addon :/
Hmmm, perhaps that's nothing, it looks like anything that matches a previous message gets retain: false anyway.
I'm now able to see the messages arriving in my analyser (seems my app was limited to 250 messages per topic so after getting the initial retained data it was saying nothing else. Upped it to 2,500 messages and now I can see everything)
But it seems to confirm my suspicion. I'm seeing the relevant lxp/cmd/BA12345/read/inputs/1 going out every 15 seconds but I'm then not seeing any other messages arriving on anything else under lxp/# in response. It's just the same standard data every 15 minutes.
I feel like there must be some step I'm missing here :/
One thing I'm a little confused by. You guys are saying that every five minutes the system is only requesting inputs 1 and 2. But that's not what I'm seeing when I monitor mine. What I'm seeing is every five minutes I see data spat out by inputs/1, inputs/2, inputs/3, and most bizarrely, inputs/all. I say bizarre for the fact that inputs/all contains everything that was included in the other three, making the other messages seem really rather redundant.
I wonder if my inverter is running different software to what has been seen previously and if this could provide clues as to why I'm not getting the behaviour you guys are describing.
When I log into the data module and check the version data it reads:
Version Information
Module Name
EMW1290
MAC
84:9D:C2:[hidden for privacy]
SDK Version
MX1290002.032.002
Software Version
ATV1.20@MX1290
Web Version
web_V1.0.0
Oh and I've just spotted this on the landing "Run State" page:
Command Mode State
Function: Disable
I wonder if that means commands of any kind are simply not allowed. I can't seem to find any setting that corresponds to this "Command Mode" to "enable" it :/
Okay... this is odd. It's working. I didn't change anything, but I restarted the container and suddenly it's all working. So clearly something had just got stuck.
I did have my electrics turned off the other day and I've restarted the inverter a couple of times. I wonder if this is an issue caused by power loss at some point causing the command listener to lock up? :/
When I've a moment to test, I'll try killing the inverter and see what happens to this, cooking right now so not a good time to do that though.
I have not been able to reproduce the issue I was having at all. It just seems to be all working now since restarting the container. I have gone ahead and closed the other ticket for now. Please do try restarting your container though if you have the same issue of commands not being respected. In the meantime, the automation suggested above by Beenay25 does indeed work perfectly now.
I might be misinterpreting what you're asking, but the following automation worked for me :
alias: Poll Battery State 1 & 2 trigger: - platform: time_pattern seconds: /15 action: - service: mqtt.publish data: topic: lxp/cmd/<YOUR INVERTER ID>/read/inputs/1 retain: true - service: mqtt.publish data: topic: lxp/cmd/<YOUR INVERTER ID>/read/inputs/2 retain: true mode: singleIt'll send out a MQTT request to the router every 15 seconds and gets a response sent out.
Thanks, this made my life easier
@apbarratt So I was a bit puzzled by this automation too. While I do get MQTT responses on the inputs/1 and inputs/2 topics, I don't get any updates at the configured interval in the HA entities — they still keep coming at 5 minute intervals when Luxpower portal queries everything.
It seems to be by design, since HA integration in lxp-bridge ties into the inputs/all topic, which only publishes after inputs/1, inputs/2 and inputs/3 were queried.
So if I understand this right, this isn't really a workaround — either input/3 needs to be queried often too or partial saves/publishes to all should be allowed.
Sorry, i just wanted to clarify, I'm wondering if its possible to pull data more frequently as the lux app does in local connect mode.
The automation above, is this also triggering updates to be sent to lux? I don't expect them to be too happy about being thrown data every 15 seconds.
Sorry, i just wanted to clarify, I'm wondering if its possible to pull data more frequently as the lux app does in local connect mode.
The automation above, is this also triggering updates to be sent to lux? I don't expect them to be too happy about being thrown data every 15 seconds.
Yes it does. The inverter sends all its packets to every connected client, no matter who requested that data. So Luxpower servers will see it (and presumably log it) if you request it every few seconds. How much of a problem this is, who knows :)
To get deeper into the detail, they only seem to log a row to their database when they see the third data packet (the one we call inputs/3. So if you only request 1 & 2 every 15 seconds (which have most of the interesting details anyway, 3 only has things that change very rarely), they won't log it - this is actually what the local app does. When you have the local app running they're still seeing the 1 & 2 packets, but no 3, so no logging happens.
On a broader note I'd suggest considering firewalling the inverter off so it has no Internet connectivity anyway, a few people are doing that successfully and we can still monitor it via the local heartbeats option.
So not sure if you are aware of this but it was posted in the Facebook Lux Power Tek Owners Group - UK
I would be careful in bumping up the refresh to high unless until all is understood
Indeed, it is quite ridiculous but it's pushed me to kill off the Chinese server access on my dongle and it's been running perfectly for over 12 hours now, confirming that lxp-bridge's heartbeats are indeed keeping things ticking along nicely.
Before
After
Indeed, it is quite ridiculous but it's pushed me to kill off the Chinese server access on my dongle and it's been running perfectly for over 12 hours now, confirming that lxp-bridge's heartbeats are indeed keeping things ticking along nicely.
I've done this and it works well. It's a shame as the LXP app is occasionally useful.
Also see https://github.com/celsworth/lxp-bridge/discussions/213 for more discussion about this.
The official post is a bit ambiguous, I'm not sure if they mean us or not with "Home Assist App", and presumably "ping settings" means the update frequency. But yes I definitely agree that keeping all the data local is beneficial for multiple reasons - ideally we'd aim to replace the LXP app entirely and be able to do everything locally.
Personally I think it's a bit of a poor show that they must have sold thousands, tens of thousands (or more) inverters yet can't spin up some server infrastructure to cope with a few hundred (at best, surely?) users sending more frequent data updates than they bargained for. But I am not surprised, it is China after all.
Indeed. I used it the other day to tweak the schedule after Octopus increased my cheap electric hours. I think that's doable through bridge too, but it was quicker to just spin up the app, though it was the first time in a while. It would have been nicer to find a toggle to turn the Chinese server off or on instead of having to delete it. I guess I'll just need to come back here and check my screenshot should I ever need it again 😅
I only keep the app in case, for some reason, I can't get through to my VPN ('cos some numpty has blocked it on their public wifi) - lol. However, all this talk of closing down APIs is just scaremongering - there's a very simple fix if they could be bothered.
The only thing I'm missing in my setup is some decent data logging - anyone got a decent node-red setup for squirting into influx?
Kind Regards John
On Thu, 26 Oct 2023 at 11:28, Andy Barratt @.***> wrote:
Indeed. I used it the other day to tweak the schedule after Octopus increased my cheap electric hours. I think that's doable through bridge too, but it was quicker to just spin up the app, though it was the first time in a while. It would have been nicer to find a toggle to turn the Chinese server off or on instead of having to delete it. I guess I'll just need to come back here and check my screenshot should I ever need it again 😅
— Reply to this email directly, view it on GitHub https://github.com/celsworth/lxp-bridge/issues/113#issuecomment-1780846304, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHTIDKIH5ZA6SBBIDIXQKDYBI3NJAVCNFSM6AAAAAASI7YPH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQHA2DMMZQGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
The only thing I'm missing in my setup is some decent data logging - anyone got a decent node-red setup for squirting into influx?
You've seen lxp-bridge can do this directly? https://github.com/celsworth/lxp-bridge/wiki/Configuration#influx
Yes, Chris - somehow unable to make it work though.
Kind Regards John
On Sat, 28 Oct 2023 at 00:35, Rob Aldred @.***> wrote:
What's the benefit of logging to influx over what home assistant provides?
— Reply to this email directly, view it on GitHub https://github.com/celsworth/lxp-bridge/issues/113#issuecomment-1783613139, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHTIDKDBRF4ISXHDLEFKRLYBRAMDAVCNFSM6AAAAAASI7YPH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTGYYTGMJTHE . You are receiving this because you commented.Message ID: @.***>
Yes it does. The inverter sends all its packets to every connected client, no matter who requested that data. So Luxpower servers will see it (and presumably log it) if you request it every few seconds. How much of a problem this is, who knows :)
@celsworth Do they still receive the packets if you configure lxp-bridge to connect directly to the dongle AP, e.g. 10.10.10.1? Similar to using Local Monitor in the app.
