custom-components
first pass
This PR is an attempt to make the integration more resilient against Nordpool API errors, by retrying on HTTP errors and missing/wrong data with exponential retries.
With the version from yesterday there is a bug that it keeps the same prices for tommorow. Today 0.171, 0.179, 0.175, 0.173, 0.175, 0.116, 0.241, 0.32, 0.348, 0.329, 0.252, 0.234, 0.223, 0.209, 0.148, 0.132, 0.093, 0.132, 0.27, 0.306, 0.306, 0.277, 0.267, 0.277 Tomorrow 0.171, 0.179, 0.175, 0.173, 0.175, 0.116, 0.241, 0.32, 0.348, 0.329, 0.252, 0.234, 0.223, 0.209, 0.148, 0.132, 0.093, 0.132, 0.27, 0.306, 0.306, 0.277, 0.267, 0.277
Raw today and Raw tomorrow have both same data, same timestamps, same values.
2022-04-02 10:37:35 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform nordpool is taking over 10 seconds. .... 2022-04-02 10:41:26 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: sensor.nordpool
Starts already for 4 minutes. something is wrong there...
Please enable the debug log and the log. The sensor can take a long time depending on the issues it has
The integration made 69 requests to finally get data. i think it fails if prices are requested before midday (when new prices are published) See log attached. nordpool-fix-1_full.log
@yozik04 Thanks for the debug log! It was really helpful, the problem was that the retry method considers all "falsy" as a request for retrying. Ill fix that :)
The method you used to make nordpool library async is very scary. I'd better throw a PR to add async to that library. =)
BTW. Take a look to the latest version of that library. It already has async support.
Ahh, you have just made a copy of that async version and modified... Ok, your code, your rules...
@yozik04 The API wasn't async when I made this, I was the one that sent an upstream PR to make it async. I kept my code to make sure that it worked. See PR https://github.com/kipe/nordpool/pull/8 and https://github.com/kipe/nordpool/pull/11 If you have a better way to make this async while supporting multiple async HTTP libraries, feel free to send a PR
@yozik04 I think it's ready to be merged. Can you test and leave it running until the prices for Monday should be available?
@yozik04 thats intended. The api stores/get the data in cet. So if you requested data area in another time zone you wouldn’t get the correct prices. That’s why we request data for 3 days and extract what we need.
Found infinty invalid data in area SYS
Why it checks SYS if I am interested in EE only?
Looks like in my region prices appear at 15:00 UTC+3. For some reason it started requesting at 14:00, which is 11 UTC.
~~But overall it works.~~ Actually not. See below.
Ahh, my father has restarted HA in the morning. He told it showed today and tomorrow same prices...
@yozik04 can you post the log? I want to check why it started requesting data then
Sure. Here you go.
yesterday evening + todays morning till 8 am nordpool-2022-04-02_03.log
todays 8 am+ nordpool-2022-04-03.log
@yozik04 I couldn't find any obvious reason for why tomorrows data wasn't cleared. I have added some better logging and fixed a potential race condition. Please update and let me know if you have any issues.
I will drop an idea here how I would do the whole thing.
I'd create a class that would hold all fetched valid values (yesterday, today, tomorrow) in a single dictionary (datetime as a key). Let's name it PriceData That class would have methods:
get_todays_data- would scan the dictionary and output todays's pricesget_tomorrows_data- would scan the dictionary and output tomorrow's priceshas_todays_data- would tell if dictionary has todays datahas_tomorrows_data- would tell if dictionary has tomorrows dataclear_expired- would scan the dictionary and drop data that is already expiredstore_data- injects the data to the dictionary (actually a nice place to validate the data as well, and inject only valid values)get_current_price- probably it is also needed, you know better.
Code out of PriceData class would query nordpool for today's data only if PriceData.has_todays_data returns False.
Same with tomorrow but should also verify that time is adequate for the request. If fetch was successful and data is valid we can still reuse has_todays_data, has_tomorrows_data
Every day something needs to call PriceData.clear_expired as well (probably run it before every request to nordpool).
That would be my design. I would even cover PriceData with some unit tests. But I do not really want to go and change your code. =) No offence.
I will drop an idea here how I would do the whole thing. ` I'd create a class that would hold all fetched valid values (yesterday, today, tomorrow) in a single dictionary (datetime as a key). Let's name it PriceData That class would have methods:
Thanks for your suggestion, that is pretty much how this is designed today with some small changes to be able to handle currencies and we also need to clear the sensors data in addition to NordpoolData as we can't do IO in a property.
get_todays_data- would scan the dictionary and output todays's pricestodayget_tomorrows_data- would scan the dictionary and output tomorrow's pricestomorrowhas_todays_data- would tell if dictionary has todays data | Does not existhas_tomorrows_data- would tell if dictionary has tomorrows data | Does not existclear_expired- would scan the dictionary and drop data that is already expirednew_day_cbstore_data- injects the data to the dictionary (actually a nice place to validate the data as well, and inject only valid values)_updateget_current_price- probably it is also needed, you know better. | Cant be handle here, must be handled in sensorCode out of PriceData class would query nordpool for today's data only if
PriceData.has_todays_datareturnsFalse. Same with tomorrow but should also verify that time is adequate for the request. If fetch was successful and data is valid we can still reusehas_todays_data,has_tomorrows_dataEvery day something needs to call
PriceData.clear_expiredas well (probably run it before every request to nordpool).new_day_cbThat would be my design. I would even cover PriceData with some unit tests. But I do not really want to go and change your code. =) No offence.
I'll accept every PR that makes the sensor more reliable as long as none of the current features are missed. 😊
At night both day prices became unavailable with new version. Will send logs later.
Thanks, that was helpful. 👍 The server was restarted after midnight so there isn't any available data for tomorrow, it will try one then, stop. However, it also did a new HTTP call to get today's data. There were invalid values in FI results, so the requests for retried over and over until the setup of the senor failed. We validate on currency as the NordpoolData don't know what data the user wants