zaptec icon indicating copy to clipboard operation
zaptec copied to clipboard

Error 429 – Too Many Requests

Open davee640 opened this issue 10 months ago • 9 comments

Hello, I am encountering an issue with the Zaptec integration for Home Assistant. An error message appears: "Error 429 – Too Many Requests." We have 16 charging stations in the installation, and synchronization is being refused. Do you have a solution?

Thank you

David

davee640 avatar Feb 18 '25 22:02 davee640

There is a limit on how frequent requests can be sent to Zaptec cloud. For many chargers, the polling interval (default 60 seconds) must be reduces. See #158

sveinse avatar Feb 19 '25 01:02 sveinse

Hello, Thank you for your response. I have already tried several intervals, but the result is not good. It indicates that the entities are not available. Here are several pictures Thank you in advance for your help.

davee640 avatar Apr 06 '25 08:04 davee640

How long intervals did you test with? When you get 429 Too many requests its the Zaptec cloud that refuses to deliver data. The only real fix to this would be to reduce the request frequency, as we have no control over the limits which Zaptec enforce on their interfaces.

Are you able to dump the diagnostics? See the front page readme for a description how.

sveinse avatar Apr 06 '25 17:04 sveinse

Since there is no option to configure the poll interval, this is just a quick guide to do it without modifying the code.

First, disable polling.

  • Go to Settings -> Devices -> Zaptec and click the three dots on the right, next to number of devices and entities.
  • Click system options
  • Disable polling

Image

Next, create a simple automation to poll based on a schedule.

  • Settings -> Automations -> Create Automation
  • As trigger, you can for instance set it to run every 2 minutes or whatever suits your needs, and that will not cause any more 429s. The period required depends on your number of devices. 2 minutes seems to work for my installation.

Image

  • As action, you select "Home Assistant Core Integration: Update Entity Image
  • As "Entity to update", just select any entity that the Zaptec integration provides (any sensor or binary sensor)

You do need to add all the entities. Just one single entity from any of the devices will suffice to poll all the devices at the set interval.

Olen avatar May 09 '25 13:05 Olen

Hello, Zaptec is here.

We do have a rate limit of 10 requests per second per user. There is a high chance the frequency of the requests is too high. We do recommend doing two things:

  1. Use exponential backoff retry.
  2. Use jitter on the requests.

https://pypi.org/project/retry/ Is a good library that has built in policies.

thecoldwine avatar May 14 '25 08:05 thecoldwine

Maintainer here.

Thank you @thecoldwine, I will take a look at this.

A wish from me were that more services and datapoints were updated via the push mechanism (ServiceBus). It would reduce the need for continuous and regular polling. That would in turn decrease the latencies for the user and lessen the request load on Zaptec cloud. My 2 cents.

sveinse avatar May 14 '25 09:05 sveinse

@sveinse we are looking into enabling a lightweight push mechanism for non-enterprise customers (we do have Service Bus for enterprise / b2b integrations).

I have no details to share on a specific technology or timelines, but it is definitely on our radar.

thecoldwine avatar May 14 '25 09:05 thecoldwine

Hello, Zaptec is here.

We do have a rate limit of 10 requests per second per user. There is a high chance the frequency of the requests is too high. We do recommend doing two things:

  1. Use exponential backoff retry.
  2. Use jitter on the requests.

https://pypi.org/project/retry/ Is a good library that has built in policies.

There is already exponential and jitter implemented in the code. But it only does this when a request fails. So with this in mind, is there anything that's missing?

https://github.com/custom-components/zaptec/blob/6734c361bef46c054273598a5c609bb9f23ae307/custom_components/zaptec/api.py#L892-L899

sveinse avatar May 14 '25 19:05 sveinse

@sveinse if it retries on 429 it is enough.

thecoldwine avatar May 14 '25 19:05 thecoldwine

With reference to #177 there are users getting into problems due to 429. I think this can be related to installations with more than one charger. I'm very curious to understand the mechanism why it starts to fail, as we do have a system for exponential back-off when errors occur.

Open technical questions / follow-up items

  1. Is it bad design to fire off requests until 429 is received and then back off, or should the code keep within limits on its own?
  2. All requests from a poll update are timed from _async_update_data() in __init__.py. Can updates take so long they start overlapping with the previous, which compound the problem? What about the timeout guard?
  3. I think we need a built-in calculator that recommends the minimum poll interval for useser with many chargers. The default value clearly doesn't scale well

sveinse avatar May 18 '25 20:05 sveinse

As I wrote in https://github.com/custom-components/zaptec/issues/177, I changed the code in my Installation and add a await asyncio.sleep(0.2) before every self._account._request, but I do not think, that this is a good design ;-) ... but at the moment, I have freedom with our 10 chargers. There is no 429 anymore and it is responsive enough for me.

Image

I let you work on a solution, that respects the API restriction with max 10 requests per second, documented here and in an bit more elegant way, I have done it.

Edit 23.5.2025: View back the last 5 days since my code change. almost always on. nearly no interruption.

Image

sofa74surfer avatar May 18 '25 20:05 sofa74surfer

We have an ongoing effort to adopt to the Zaptec API fair use policy, where they suggest a far lower frequency of updates that what this integration is currently using. I believe that is the root cause why we are hitting the limits with many chargers.

The issue #188 has been created to capture this work.

sveinse avatar Jul 11 '25 10:07 sveinse

We are in the finalizing touches of releasing a beta version of 0.8. We have been working very hard to fix the polling issues and the 529 issues you have experienced. If you're feeling adventurous the version in master is ready to test. If you want to wait until there is something installable from HACS and with proper release notes, then stay tuned for the next couple of days.

We are very keen on getting feedback from the usage of it, especially for those of you with very many chargers. So you would help us to help you if you test this and give us feedback.

sveinse avatar Jul 23 '25 23:07 sveinse

v0.8b1 beta is out: https://github.com/custom-components/zaptec/releases/tag/v0.8b1

sveinse avatar Jul 24 '25 20:07 sveinse

@davee640 @sofa74surfer v0.8.2 is out. Can you please test if this fixes your issue? I'd like to close this issue if I'm not hearing back from you, as this issue is old.

sveinse avatar Aug 23 '25 11:08 sveinse

@sveinse now the problem is solve !!! ❤️

I have 20 charges zaptec pro and all work good !

Thank you very much !

davee640 avatar Aug 24 '25 10:08 davee640

looks great @sveinse, I have installed the release yesterday, 10 Chargers. Everything works fine. Sorry, I was in holiday some weeks and could not contribute the pre-release testing.

sofa74surfer avatar Aug 24 '25 11:08 sofa74surfer

Thank you both. Closing issue.

sveinse avatar Aug 24 '25 13:08 sveinse