core icon indicating copy to clipboard operation
core copied to clipboard

503 or 429 error from Hue bridge - its probably overloaded. Giving up.

Open jcontrady opened this issue 1 year ago • 6 comments

The problem

When turning on a hue light from an entities card on a dashboard, I received the error message below and the light does not turn on. Light states in HA update to display correctly when they are changed from the native Hue App. Automations issuing light.turn_on or light.turn_off actions also fail.

Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.

A few attempts to restore the Hue integration function

  1. Reload the integration

Failed to reload

  1. Cleaned up the Hue bridge. Then reload the integration

Failed to reload

  1. Removed and added the Hue integration.

Hue bridge auto-detected and added all expected devices, services, and entities. Same error message when attempting to turn on the light again.

Rolled back to 2024.9.1 and the Hue integration works again.

What version of Home Assistant Core has the issue?

core-2024.9.2

What was the last working version of Home Assistant Core?

core-2024.9.1

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Philips Hue

Link to integration documentation on our website

https://www.home-assistant.io/integrations/hue/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-09-24 16:16:43.267 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139661446639920] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call
    return await task(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state
    await self.update(id, update_obj)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update
    await self._bridge.request("put", endpoint, json=data)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
    raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 626, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 254, in async_turn_on
    await self.bridge.async_request_call(
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
    raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-09-24 16:19:50.375 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:50.377 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:50.615 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139661446639920] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call
    return await task(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state
    await self.update(id, update_obj)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update
    await self._bridge.request("put", endpoint, json=data)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
    raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 626, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 254, in async_turn_on
    await self.bridge.async_request_call(
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
    raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-09-24 16:19:51.138 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 3.25 seconds
2024-09-24 16:19:52.381 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:52.387 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:53.077 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.5 seconds
2024-09-24 16:19:53.582 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.75 seconds
2024-09-24 16:19:54.337 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.0 seconds
2024-09-24 16:19:54.576 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:54.576 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:54.579 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 3.5 seconds
2024-09-24 16:19:55.023 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 5.25 seconds
2024-09-24 16:19:55.343 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.25 seconds
2024-09-24 16:19:56.582 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:56.582 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:56.598 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.5 seconds
2024-09-24 16:19:58.145 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.75 seconds
2024-09-24 16:19:58.186 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 3.75 seconds
2024-09-24 16:19:58.588 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:58.588 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:59.900 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 2.0 seconds
2024-09-24 16:20:00.279 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 5.5 seconds
2024-09-24 16:20:00.684 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:20:00.685 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:20:01.905 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 2.25 seconds
2024-09-24 16:20:02.034 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 4.0 seconds
2024-09-24 16:20:02.692 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:20:02.692 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:20:02.976 INFO (MainThread) [aiohue.v2[X.X.X.X]] Connection to bridge closed.
2024-09-24 16:20:04.377 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.5 seconds
2024-09-24 16:20:04.789 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:20:04.789 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:20:04.883 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.75 seconds

Additional information

No response

jcontrady avatar Sep 26 '24 00:09 jcontrady

Hey there @balloob, @marcelveldt, mind taking a look at this issue as it has been labeled with an integration (hue) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of hue can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign hue Removes the current integration label and assignees on the issue, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)


hue documentation hue source (message by IssueLinks)

home-assistant[bot] avatar Sep 26 '24 00:09 home-assistant[bot]

I'm not sure if you welcome "me too!" messages, but I'm having a fault of the exact same description which led me to this issue.

I will note that WRITING commands from HA to the Hue hub fails with the 'Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.' message provided by jcontrady

However, READING from the Hue hub into HA still seems to work-- my motion sensors still happily report events, temperature, and light levels, and if I switch a light on with some other tool (the official Android app, or Hue Essentials, or Hue Menu Bar on the Mac, or Hoppla on Linux), the device's status change is reflected in HA.

The Hue Hub is a BSB002 running firmware 1.66.1966117030, which hueblog.de indicates was released sometime around 23 September, corresponding roughly with the timing of this issue's presentation. I bet Philips b0rked something.

Restoring from my July backup did not help. Deleting the Hue integration and re-pairing with the hub did not help. If you want any logs or troubleshooting from me, I'm happy to help, otherwise I'll go away.

theodric avatar Sep 30 '24 20:09 theodric

Hi @theodric, Thanks for mentioning the Hue hub model and firmware. I have the same BSB002 on firmware 1.66.1966117030 and noticed the same behavior that reading devices into HA works. Not sure I grabbed the correct logs so if you want to add yours I can't imagine it would hurt.

jcontrady avatar Oct 01 '24 18:10 jcontrady

I'm also having this same issue, started for me in core 2024.10.01. My Hue bridge is also on the same firmware version.

ardichoke avatar Oct 07 '24 11:10 ardichoke

Same problem here with firmware 1.67.1967054010 What is strange is that I have a second hub with the same firmware but this one is still working

albaintor avatar Oct 08 '24 05:10 albaintor

Same problem, started for me in core 2024.10.01. Hue Bridge model BSB002, SW Version: 1.66.1966117030 from 24.09.2024.

Removing the Hue integration from HA and re-adding it helps, but after a few hours it stops working again.

LOG from Hue Bridge Integration in HA: `homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up. 2024-10-09 08:44:48.770 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 2.0 seconds 2024-10-09 08:44:49.082 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 6.0 seconds 2024-10-09 08:44:50.088 DEBUG (MainThread) [aiohue.v2[192.168.50.126].events] Connected to EventStream 2024-10-09 08:44:50.092 DEBUG (MainThread) [aiohue.v2[192.168.50.126].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds 2024-10-09 08:44:50.326 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 4.5 seconds 2024-10-09 08:44:50.654 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139940847481136] Unexpected exception Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call return await task(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state await self.update(id, update_obj) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update await self._bridge.request("put", endpoint, json=data) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/init.py", line 184, in request raise BridgeBusy( aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service response = await hass.services.async_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/adaptive_lighting/hass_utils.py", line 62, in service_func_proxy await existing_service.job.target(call) File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call single_response = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/light/init.py", line 628, in async_handle_light_on_service await light.async_turn_on(**filter_turn_on_params(light, params)) File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 255, in async_turn_on await self.bridge.async_request_call( File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call raise HomeAssistantError(msg) from err homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up. 2024-10-09 08:44:50.779 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 2.25 seconds 2024-10-09 08:44:51.143 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139940847481136] Unexpected exception Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call return await task(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state await self.update(id, update_obj) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update await self._bridge.request("put", endpoint, json=data) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/init.py", line 184, in request raise BridgeBusy( aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.`

petos-hue avatar Oct 09 '24 07:10 petos-hue

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯_(ツ)_/¯

theodric avatar Oct 17 '24 20:10 theodric

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯(ツ)

Did it stay working? I could get mine working again just by restarting Home Assistant core, but it would stop working again after a little while with the same errors. Haven't tried 10.2 or 10.3 yet though.

ardichoke avatar Oct 18 '24 18:10 ardichoke

Tried 2024.10.3, issue still happens there. Downgraded again to 2024.9.3 and it went back to working without issue.

ardichoke avatar Oct 21 '24 01:10 ardichoke

Same issue here with 2024.10.3. Downgrade to 2024.9.3 and works without issue. Using this command: ha core update --version=2024.9.3 https://community.home-assistant.io/t/how-to-downgrade-home-assistant/179735

cmnitz avatar Oct 24 '24 16:10 cmnitz

This apparently is a widespread issue, added a new generic Tuya color bulb to my Hue bridge this morning, commands were sporadic and the bridge became non-responsive at times. Downgrade HA Core to 2024.9.3 and works without issue, didn't even have to restart the Hue bridge.

TC1666 avatar Oct 27 '24 14:10 TC1666

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯(ツ)

Did it stay working? I could get mine working again just by restarting Home Assistant core, but it would stop working again after a little while with the same errors. Haven't tried 10.2 or 10.3 yet though.

I think it's been about ten days now and it's still working (again) for me post-power outage. After it magically started working, I upgraded my HAOS aarch64 VM to core 2024.10.4, Supervisor 2024.10.3, and OS 13.2, and the Hue integration still functions.

theodric avatar Oct 27 '24 15:10 theodric

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯(ツ)

Did it stay working? I could get mine working again just by restarting Home Assistant core, but it would stop working again after a little while with the same errors. Haven't tried 10.2 or 10.3 yet though.

I think it's been about ten days now and it's still working (again) for me post-power outage. After it magically started working, I upgraded my HAOS aarch64 VM to core 2024.10.4, Supervisor 2024.10.3, and OS 13.2, and the Hue integration still functions.

You seem to have gotten really lucky. I powered my haos VM off and unplugged my hue bridge, then powered both back up. After a few hours, it went right back to a non functional state again on 2024.10.4.

ardichoke avatar Oct 27 '24 22:10 ardichoke

I cant reproduce this issue at all. The only way to trigger this is if there are MANY commands to the Hue hub. At some point its (very limited) webserver will start denying requests.

  • Do you have other apps connected to the Hue bridge like Hue Essentials or node Red ? If so, try shutting them down for a while

  • Do you have any integrations or automations that do many commands to Hue lights such as Adaptive lightning ? If so, try disabling that for a while to test.

This may as well have some other external cause such as an HTTP connection not being closed properly etc. That will need to be closely looked at.

marcelveldt avatar Oct 28 '24 08:10 marcelveldt

Obviously, I cannot speak for anyone else here, but I can answer these for my experience.

* Do you have other apps connected to the Hue bridge like Hue Essentials or node Red ?
  If so, try shutting them down for a while

I only have two things talking to Hue, Home Assistant and Google Assistant. I rarely use Google Assistant to interact with my Hue lights, but even when Home Assistant has been throwing these "giving up" errors, I have been able to control my Hue lights via Google Assistant (and via the Hue app). This certainly makes it seem like the problem is on the HA side.

* Do you have any integrations or automations that do many commands to Hue lights such as Adaptive lightning ?
  If so, try disabling that for a while to test.

No. The only interactions that HA does with Hue is turning on Hue scenes and turning Hue lights off. Nothing more.

This may as well have some other external cause such as an HTTP connection not being closed properly etc. That will need to be closely looked at.

For me, at least, this is solely happening when I update HA core to 2024.10.1 or later. If I'm running 2024.9.3 or earlier, I have no issues. I don't know if something changed in the core that would impact requests sent to the Hue bridge, or if this is pure coincidence. I can say I've upgraded HA core to each of the 2024.10 versions when they came out to test and every time this started happening, after which I would downgrade to 2024.9.3 and the integration would work as expected again, so it definitely seems to be consistent and repeatable in my install at least.

ardichoke avatar Oct 28 '24 13:10 ardichoke

Thanks for the additional info - I'll have to investigate if there have been any changes to our http client then (aiohttp).

marcelveldt avatar Oct 28 '24 17:10 marcelveldt

If there's anything I can provide to help troubleshoot this, please let me know since I can reliably cause the issue to occur.

ardichoke avatar Oct 28 '24 21:10 ardichoke

I just wanted to chime in and say that I also ran into this issue. Hub is running 1.67.1967054020 and HA is on 2024.10.2.

Best I can tell is that it started 3 days ago based off the last time one of my automations turned on a light.

As with @theodric, reading the state from the hub worked, but writing did not. A reboot of the hub seems to have fixed it for now at least.

Here is the log from HA, but it doesn't look to be anything particularly useful

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
response = await hass.services.async_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/homeassistant/__init__.py", line 151, in async_handle_turn_service
await asyncio.gather(*tasks)
File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1021, in entity_service_call
raise result from None
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1610, in async_request_call
return await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
result = await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 628, in async_handle_light_on_service
await light.async_turn_on(**filter_turn_on_params(light, params))
File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 255, in async_turn_on
await self.bridge.async_request_call(
File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-10-29 13:55:17.553 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 559, in _handle
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 210, in forwarded_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
response = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
result = await handler(request, **request.match_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/diagnostics/__init__.py", line 280, in get
data = await info.config_entry_diagnostics(hass, config_entry)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/hue/diagnostics.py", line 23, in async_get_config_entry_diagnostics
return await bridge.api.get_diagnostics()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 276, in get_diagnostics
full_state = await self.request("get", "clip/v2/resource")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.```

k-masters avatar Oct 29 '24 18:10 k-masters

Any updates on this? I upgraded to 2024.11.01 and had this happen again afterwards. I'm wondering if I should try removing and adding my hue bridge again, but its really annoying having to set up my automations again...

Based on my own experiences only, it looks like aiohttp got updated from 3.10.5 to 3.10.8 between the working and problematic releases. This is the diff between those versions: https://github.com/aio-libs/aiohttp/compare/v3.10.5...v3.10.8

ardichoke avatar Nov 13 '24 14:11 ardichoke

Any updates on this? I upgraded to 2024.11.01 and had this happen again afterwards. I'm wondering if I should try removing and adding my hue bridge again, but its really annoying having to set up my automations again...

FWIW I upgraded from 2024.9.3 to 2024.11.1 and haven’t had the issue since.

cmnitz avatar Nov 14 '24 23:11 cmnitz

Still persists for me in 2024.11.2. I grabbed a debug log today of it happening, but I don't think there's anything particularly useful in it. Just a lot of this:

2024-11-19 17:41:51.144 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.5 seconds
2024-11-19 17:41:51.250 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.250 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:51.496 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.496 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:51.500 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.500 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:51.729 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.5 seconds
2024-11-19 17:41:51.767 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.768 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:52.649 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 3.5 seconds
2024-11-19 17:41:53.364 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:53.365 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:53.678 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:53.701 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:53.780 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:55.480 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:56.260 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 3.75 seconds
2024-11-19 17:41:56.754 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.75 seconds
2024-11-19 17:41:57.335 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.75 seconds
2024-11-19 17:42:00.017 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.0 seconds
2024-11-19 17:42:02.510 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.0 seconds
2024-11-19 17:42:03.091 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.0 seconds
2024-11-19 17:42:04.023 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.25 seconds
2024-11-19 17:42:08.279 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.5 seconds
2024-11-19 17:42:08.515 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.25 seconds
2024-11-19 17:42:09.097 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.25 seconds
2024-11-19 17:42:12.785 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.75 seconds
2024-11-19 17:42:14.773 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Error while sending keepalive: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-11-19 17:42:15.350 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Error while sending keepalive: 25 requests to the bridge failed, its probably overloaded. Giving up.

If desired, I can provide the full log, but I don't see anything all that useful.

I do see occasional 503 or 429 messages throughout the day, but normally after a retry or two it is successful. Eventually, something tips and it starts hitting the 25 retry, giving up state at which point it does not ever recover.

I also tried rebooting the Hue Bridge, to see if that would restore functionality, but even after that Home Assistant is still doing the same thing, which further suggests that the problem is on the Home Assistant side of the equation.

It makes me wonder if the retries are too aggressive, or if something is continuing to send requests to the bridge in the background which eventually leads to a cascading failure.

ardichoke avatar Nov 19 '24 23:11 ardichoke

I don't have good evidence to support this theory, because I didn't notice things were working straight away, but I also had this issue for a while.

I separately noticed that my HA wall time was wrong by a few minutes, which I resolved by installing ntpd on my docker host.

Some time later I noticed Hue devices were all showing up normally again. Could a time difference between the Hue bridge & HA cause something like this?

I don't really want to switch back to systemd-timesyncd which doesn't keep my system clock in time just to confirm this, sorry 😅

funkyHat avatar Nov 19 '24 23:11 funkyHat

Today I was (finally!) able to reproduce this issue after adding a lot more lights to one single Hub. So this seems to be a combination of factors but it is has to do with the amount of resources (lights/devices/scenes etc) on the Hue hub - makes sense as that will also increase the json payloads. This may as well be as simple as a timeout or message size limit. In my case, a restart of the Hue bridge resolved the issue. I'll poke more in to this tomorrow.

Question to you all is how many resources you have on the hue bridge ? So lights, zones, rooms, scenes and remotes etc. Give or take, doesnt have to be precise. Just to have an indication if this only happens on large setups.

marcelveldt avatar Nov 26 '24 19:11 marcelveldt

I do have a large setup : about 50 bulbs and 15 devices

albaintor avatar Nov 26 '24 19:11 albaintor

I have a pretty small Hue setup... 8 bulbs, 4 rooms, 1 zone, 28 scenes (this is just the "natural light" scene in each of the 4 rooms)

ardichoke avatar Nov 26 '24 20:11 ardichoke

I do have a mid-size setup with 24 lamps, 3 motion sensors and 2 dial switches. All connected to on one hub.
All in all the hue Integration in Home assistant has 30 devices, 10 services and 155 entities.

itSme1990 avatar Nov 26 '24 21:11 itSme1990

Based on my own experiences only, it looks like aiohttp got updated from 3.10.5 to 3.10.8 between the working and problematic releases. This is the diff between those versions: https://github.com/aio-libs/aiohttp/compare/v3.10.5...v3.10.8

Yeah, I noticed that as well and its actually the only change that could possibly explain the behavior. I'm currently trying to reproduce the issue with and without that aiohttp bump involved to pinpoint it. So far I was not able to reproduce with downgrading aiohttp.

marcelveldt avatar Nov 27 '24 08:11 marcelveldt

After the hub reboot way back when I made my last comment sometime last month I haven't had the issue pop back up. I'm still running HA 2025.10.2.

As far as my Hue setup, I'd say it's rather small. 10 bulbs, 1 switch, 4 rooms, and 20 or so scenes of which most are the default ones from the Hue app.

My Hue integration has 11 devices, 4 services, and 66 entities.

k-masters avatar Nov 27 '24 13:11 k-masters

I'm also experiencing this issue. Press on/off on an individual light in HA, nothing happens. Sometimes 20 seconds later the light will 'catch up' and change state.

5 devices (4 lights + bridge), 3 services, 11 entities

Via HA: Delays, issues as seen from other commenters Via HTTP PUT via Hue API / python requests.put : all works ok. I can spam it with JSON states as many times as I want and it keeps working, even during the failing HA errors.

Hue Bridge: 1.67.1967054020 HA Core: 2024.11.3

Logger: homeassistant.core
Source: core.py:2822
First occurred: 14:26:35 (5 occurrences)
Last logged: 14:27:28

Error executing service: <ServiceCall light.turn_off (c:<somerandomID>): entity_id=['light.mylight1'], params=>
Error executing service: <ServiceCall light.turn_off (c:<somerandomID>): entity_id=['light.mylight2'], params=>
Error executing service: <ServiceCall light.turn_on (c:<somerandomID>): entity_id=['light.mylight2'], params=>
Error executing service: <ServiceCall light.turn_on (c:<somerandomID>): entity_id=['light.mylight1'], params=>
Error executing service: <ServiceCall light.turn_off (c:<somerandomID>): entity_id=['light.mylight1'], params=>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call
    return await task(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state
    await self.update(id, update_obj)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update
    await self._bridge.request("put", endpoint, json=data)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
    raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 2822, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 639, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 280, in async_turn_off
    await self.bridge.async_request_call(
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
    raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.

c4bler avatar Nov 27 '24 14:11 c4bler

Hitting the same issue (I installed my hue bridge a couple days ago), my setup is rather small: 4 lamps in 1 room, and one hue remote (dimmer switch). Changes from the hue app are immediate and mirrored in HA at once, but actions from HA timeout almost always. I can test integration patches and/or provide logs as needed.

speed47 avatar Nov 27 '24 18:11 speed47