core icon indicating copy to clipboard operation
core copied to clipboard

Powerview Polling causing timeout errors

Open kingy444 opened this issue 2 years ago • 96 comments

I put some logging in the new polling implemented under https://github.com/home-assistant/core/pull/73659 This PR resolvee https://github.com/home-assistant/core/issues/70043 but has unfortunately created an issue where shades will timeout from time to time.

While the hub can process the requests the shades cannot always return a result in a timely manner. I have 5 hardwired shades but only 2 enabled for polling and still see these around every 6 hours or so.

These shades are all TDBU, which the current code would cause to poll twice however this is unrelated as testing is the same under PR to resolve that issue: https://github.com/home-assistant/core/pull/73899

image

Originally posted by @kingy444 in https://github.com/home-assistant/core/pull/73659#issuecomment-1162798204

kingy444 avatar Jun 23 '22 12:06 kingy444

If its still chatty with your fix, I think it would be fine to suppress any errors from the single polls since we should get errors from the main coordinator poll

bdraco avatar Jun 23 '22 20:06 bdraco

Hey there @trullock, mind taking a look at this issue as it has been labeled with an integration (hunterdouglas_powerview) you are listed as a code owner for? Thanks! (message by CodeOwnersMention)

Same issue here. The polling over loads the unit and then crashes it. Only fix is hard reboot of the hub.

Man I hate HD

chrisjenx avatar Jul 10 '22 14:07 chrisjenx

I have 30 shades FYI, so the unit becomes inoperable, is there a way to disable polling or change frequency?

chrisjenx avatar Jul 10 '22 14:07 chrisjenx

sigh. I guess we need an option to disable polling as well.

I'm traveling this week so I won't be able to add it too soon.

I suggest manually patching should_poll to False in the mean time.

bdraco avatar Jul 10 '22 16:07 bdraco

Yeah no rush, seems to take about a week to finally build up

chrisjenx avatar Jul 10 '22 18:07 chrisjenx

Actually you can already disable it so no code change needed

On Jul 10, 2022, at 1:27 PM, Christopher Jenkins @.***> wrote:

Yeah no rush, seems to take about a week to finally build up

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/73900#issuecomment-1179776600, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFB7CFDHZQPOUFN3P3QSVLVTMIXXANCNFSM5ZUF2Z7Q. You are receiving this because you were assigned.

bdraco avatar Jul 11 '22 00:07 bdraco

Under system options for the config entry, turn off

Enable polling for updates.

If Home Assistant should automatically poll Hunter Douglas PowerView entities for updates.

On Jul 10, 2022, at 7:10 PM, J. Nick Koston @.***> wrote:

Actually you can already disable it so no code change needed

<IMG_1268.jpg><IMG_1269.PNG>

On Jul 10, 2022, at 1:27 PM, Christopher Jenkins @.*** @.***>> wrote:

Yeah no rush, seems to take about a week to finally build up

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/73900#issuecomment-1179776600, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFB7CFDHZQPOUFN3P3QSVLVTMIXXANCNFSM5ZUF2Z7Q. You are receiving this because you were assigned.

bdraco avatar Jul 11 '22 00:07 bdraco

Under system options for the config entry, turn off

Enable polling for updates.

If Home Assistant should automatically poll Hunter Douglas PowerView entities for updates.

bdraco avatar Jul 11 '22 00:07 bdraco

I thought that disables polling the powerview completely?

chrisjenx avatar Jul 11 '22 15:07 chrisjenx

I have 30 shades FYI, so the unit becomes inoperable, is there a way to disable polling or change frequency?

I have only 7 shades. Since upgrading HA a few versions ago I need to restart my Powerview every few days.

Thought about opening an issue then found this one. Still trying to debug what causes it and how to reliably reproduce it.

glyph-se avatar Jul 19 '22 18:07 glyph-se

I have 30 shades FYI, so the unit becomes inoperable, is there a way to disable polling or change frequency?

I have only 7 shades. Since upgrading HA a few versions ago I need to restart my Powerview every few days.

Thought about opening an issue then found this one. Still trying to debug what causes it and how to reliably reproduce it.

It's odd, since I posted that I haven't had to reboot the controller.. I control most of my shades via scenes which are way more reliable than single control.

chrisjenx avatar Jul 19 '22 20:07 chrisjenx

I have 30 shades FYI, so the unit becomes inoperable, is there a way to disable polling or change frequency?

I have only 7 shades. Since upgrading HA a few versions ago I need to restart my Powerview every few days. Thought about opening an issue then found this one. Still trying to debug what causes it and how to reliably reproduce it.

It's odd, since I posted that I haven't had to reboot the controller.. I control most of my shades via scenes which are way more reliable than single control.

I just updated to 2022.7.6 and my PowerView hub hung right after.

I have tried manually restarting Home Assistant a few times, but the hub doesn't seem to be affected by this. Thinking back, the other times it hung could be after other updates, I haven't seen it in a while, and 2022.7.5 is quite old.

Could it be the same for you @chrisjenx ?

Shall I open a new issue for this?

Edit: Now it hung after restart, so either only sometimes, or there is a delay.

glyph-se avatar Jul 21 '22 14:07 glyph-se

My powerview hub was working fine for 19 days, I just updated to 2022.8.2 (from 2022.7.6) and it hung itself again.

glyph-se avatar Aug 09 '22 20:08 glyph-se

Yeah, it seems to happen when it sends a bunch of commands to the PowerView all at once (which I'm guessing happens after a reboot/restart when the component is reloaded) I have mine powered by PoE splitter, debating creating an automation that just reboots the PV at like 4am.

chrisjenx avatar Aug 10 '22 02:08 chrisjenx

I have a PR for the issue this ticket was originally raised for but this ticket spawned a set of users with a different issue where the PowerView Hub would lock up after an update

can any of you advise the below: 1 - does this issue still occur 2 - does the issue occur only on update ? I can’t see why it wouldn’t also occur on reboot 3 - how many hard wired powerview shades do you have. 4 - have you come across any other wierdness. I have 5 hardwired shades, all top down bottom up. The continued polling seems to put their calibration out, and I have to recalibrate every couple of weeks so I have actually set them to battery powered in the app to avoid polling all together (and the issue went away)

kingy444 avatar Oct 24 '22 08:10 kingy444

Yes still happens for me I have 34 wired shades Update is worst as it refreshes the integration that seems to over load the PV.

Best solution I found is create an automation to reboot the PV hub every few days. That seems to clear out what ever memory leak the device has.

chrisjenx avatar Oct 24 '22 13:10 chrisjenx

  1. I haven't seen it in a while, though I don't know what could have changed.
  2. I have only seen it on Home Assistant update, though I rarely do reboot otherwise. But I did try rebooting a few times without experiencing the issue
  3. None wired, 7 battery powered
  4. I have some connection issues where one out of 2-3 shades sometimes lose connection with the hub. I do believe this is due to bad wireless coverage of the powerview network. I currently have one extender, but I don't think this is enough. The 3 closest shades always work. I also have one shade stuck on FW 1.8, where all my others have FW 1.10

On Mon, 24 Oct 2022 at 10:24, kingy444 @.***> wrote:

I have a PR for the issue this ticket was originally raised for but this ticket spawned a set of users with a different issue where the PowerView Hub would lock up after an update

can any of you advise the below: 1 - does this issue still occur 2 - does the issue occur only on update ? I can’t see why it wouldn’t also occur on reboot 3 - how many hard wired powerview shades do you have. 4 - have you come across any other wierdness. I have 5 hardwired shades, all top down bottom up. The continued polling seems to put their calibration out, and I have to recalibrate every couple of weeks so I have actually set them to battery powered in the app to avoid polling all together (and the issue went away)

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/73900#issuecomment-1288625377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALKQUPETQDJBTQNFOTYS33WEZBU3ANCNFSM5ZUF2Z7Q . You are receiving this because you commented.Message ID: @.***>

glyph-se avatar Oct 25 '22 16:10 glyph-se

FWIW, I also see this from time to time, unless I turn off polling in system options. I have 33 wired shades. Regarding connectivity/coverage: I have a secondary hub and several repeaters. I haven't needed to reboot the secondary hub to resolve hass issues.

@chrisjenx does your reboot automation do anything to test whether the problem is currently manifesting, or does it just reboot every couple of days?

wez avatar Dec 08 '22 15:12 wez

To be honest I forgot to setup the power cycling, it's been a while since I've had to reboot the unit tbh, but with winter the automation keeps the shades open to increase solar gain so won't move them as much

chrisjenx avatar Dec 08 '22 18:12 chrisjenx

I have this happening regularly, I currently have to reboot my hub at least weekly, which is really annoying.

This started happening in an update made in the last few months, definitely since this https://github.com/home-assistant/core/issues/70763

I'll help debug if you let me know what to provide

Also happy to work on a fix if someone can get me up to speed on whats happening

Update

I have 3 wireless Silhouette type shades.

HD app reports Version 3.1.6 build 66895

Repeaters' firmware is at 2.0.2928

I haven't been able to determine what causes this, it could be due to rebooting HA but I dont think this is always the case. Sometimes I just come to manually move the blinds or let an automation open/close them and they dont respond. The official HD app also doesn't then affect the blinds. I have to reboot the hub to make them work again.

trullock avatar Dec 19 '22 10:12 trullock

@bdraco there are 2 seperate issues here

the one the ticket was raised for relates to the hub timing out (as I’m not responding fast enough not timedout=true from the hub)

the second is the one that probably relates to “problem with device”

I believe I have a fix for the former, and have a PR mostly finished but it takes a bit to test as the timeout cannot be forced

the latter, I have also experienced but less consistently again making it hard to troubleshoot. @trullock I don’t believe logs will help as it isn’t HA freezing, it is the HD Hub. My best guess is the hub struggles to process multiple shades returning a forced update at once. I was going to look if there is anyway to stagger the calls but don’t know if that is possible. @bdraco is it possible to do that?

Otherwise we probably need to disable polling by default @bdraco as it is highly unlikely HD do anything with the Hub with Gen 3 now out

edit: just adding it’s definitely not taking a massive amount of shades to cause this issue - I too had mine freeze with just 3 shades. I actually have all mine (including the hardwired) set to battery atm because I haven’t had much time and the hub freezing is a pain

kingy444 avatar Dec 19 '22 21:12 kingy444

I was going to look if there is anyway to stagger the calls but don’t know if that is possible. @bdraco is it possible to do that?

__init__.py:PARALLEL_UPDATES = 1
cover.py:PARALLEL_UPDATES = 1

We already have PARALLEL_UPDATES set to 1 so it should only poll one at a time.

Otherwise we probably need to disable polling by default @bdraco as it is highly unlikely HD do anything with the Hub with Gen 3 now out

It would be great if we could come up with a way to trigger the issue so we can work around it (if possible). I haven't been able to get mine to freeze.

bdraco avatar Dec 19 '22 21:12 bdraco

It might make sense to have a central polling coordinator that polls, waits 10, polls the next one, etc so it gives the hub some time between polls

bdraco avatar Dec 19 '22 21:12 bdraco

I'm not sure if it is related to any of the above issues or not, but I believe that HA sends commands to fast to the hub. I don't use scenes, but rather automations to move my blinds 2 or 3 at a time.

About 5-10% of the times only some moves. And then when I run the automation again the other blinds move. Thus I don't think it is related to any hub hanging, though it could possibly be a bad wireless connection to the hub, but I think I would have seen other issues (e.g. with the app) if that also were the case.

glyph-se avatar Dec 20 '22 06:12 glyph-se

@kingy444 the timeout cant be forced but it does happen to me fairly regularly. If you publish the fix behind a flag I can run it for a few weeks and see if theres an improvement

trullock avatar Dec 21 '22 10:12 trullock

I'm having the same issues. About 10% of the time, a 2-3 of my shades do not move. And every week or so, all shades stop working (via PowerView app or Home Assistant) and I have to reboot the hub to get it working again. I have 10 battery powered shades with 1 hub and 3 repeaters. Although a minor issue in the scheme of things, with a pregnant wife who expects the shades to be down when she needs to nap, I'm trying to find a fix to prevent a tongue lashing about my "not-so-smart home."

To anyone that created an automation, script, node red flow, or otherwise to reboot the HD hub, would you mind pasting the code? I'm relatively new with HA and do not have a coding background so I'm trying to teach myself. If not, no worries--just thought I'd ask.

downercc avatar Jan 11 '23 02:01 downercc

@downercc our sanity has been saved by doing this:

in /config/integrations, on the PowerView integration, click on the three dots and select "System Options", then turn off "Enable polling for Updates".

The consequence is that hass doesn't always know the precise state of the covers, but that's fine for our particular usage.

wez avatar Jan 11 '23 03:01 wez

@wez re consequences, if I only control the blinds via HA, there should never be a state sync issue, right? (assuming the blinds batteries arent flat, or similar, so it has always moved as instructed)

trullock avatar Jan 11 '23 09:01 trullock