core
core copied to clipboard
home-assistant
Met Office API Migration to DataHub
The problem
The Integration does not support datahub api and datapoint api is being deprecated by April 20th 2024.
There is a MetOffice API migration that creates new credentials and new baseurl - see the docs - this is likely an issue with the upstream datapoint dependency, but this would likely need at least an update to the URL constants set in that library to use the new datahub at a minimum.
At least as of today, you are unable to subscribe to the legacy datapoint API on the metoffice website, so the integration will not work for any new users (as a bonus, the error message in the config flow is not very clear if the integration already has services enabled, and when trying to add the integration without existing services, there is no clear error message/behaviour)
What version of Home Assistant Core has the issue?
core-2024.1.6
What was the last working version of Home Assistant Core?
core-2024.1.6
What type of installation are you running?
Home Assistant OS
Integration causing the issue
metoffice
Link to integration documentation on our website
https://www.home-assistant.io/integrations/metoffice
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
No response
Additional information
No response
Hey there @mrharcombe, @avee87, mind taking a look at this issue as it has been labeled with an integration (metoffice) you are listed as a code owner for? Thanks!
Code owner commands
Code owners of metoffice can trigger bot actions by commenting:
-
@home-assistant closeCloses the issue. -
@home-assistant rename Awesome new titleRenames the issue. -
@home-assistant reopenReopen the issue. -
@home-assistant unassign metofficeRemoves the current integration label and assignees on the issue, add the integration domain after the command. -
@home-assistant add-label needs-more-informationAdd a label (needs-more-information, problem in dependency, problem in custom component) to the issue. -
@home-assistant remove-label needs-more-informationRemove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.
(message by CodeOwnersMention)
metoffice documentation metoffice source (message by IssueLinks)
![home-assistant[bot] avatar](https://avatars.githubusercontent.com/in/97978?v=4 "Published by a pub.dev verified publisher"){kind=small}
You're probably aware of this, but just in case:
https://github.com/home-assistant/core/pull/108163
With the integration already being disabled from 2024.2 due to Python incompatibillty, seems like it will need updating to the new DataHub API anyway if it's ever going to get back into HA.
Absolutely. And z when last I looked there wasn't a library in PyPi that supported Data hub, let alone supported Python 3.12, which makes this a reasonably-size endeavour.
And, sadly, one I simply don't have time for 😞
Ian Harcombe
On Thu, 1 Feb 2024, 17:01 ResteNarquois, @.***> wrote:
You're probably aware of this, but just in case:
https://github.com/home-assistant/core/pull/108163 http://url
With the integration already being disabled from 2024.2 due to Python incompatibillty, seems like it will need updating to the new DataHub API anyway if it's ever going to get back into HA.
— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/109301#issuecomment-1921787059, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6AYWMAJNZZH7AWI7CKTYRPDABAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRG44DOMBVHE . You are receiving this because you were mentioned.Message ID: @.***>
I stand corrected! Although one release in 2021, with no link to a repository, doesn't inspire the greatest confidence...
Ian Harcombe
On Sat, 3 Feb 2024, 20:45 danuals, @.***> wrote:
https://pypi.org/project/metoffice-datahub/
— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/109301#issuecomment-1925451207, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6A6W74LJFLHQ247S6W3YR2OXRAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGQ2TCMRQG4 . You are receiving this because you were mentioned.Message ID: @.***>
https://github.com/MetOffice/weather_datahub_utilities
https://pypi.org/project/metoffice-weather-datahub/
any good?
After upgrading to core-24.2.0 the integration fails to load with a Python version error.
Logger: homeassistant.setup
Source: setup.py:221
First occurred: 9:29:25 AM (1 occurrences)
Last logged: 9:29:25 AM
Setup failed for 'metoffice': Dependency is disabled - Integration library not compatible with Python 3.12
https://github.com/MetOffice/weather_datahub_utilities
https://pypi.org/project/metoffice-weather-datahub/
any good?
That is the exemplar code, and it looks like a few adaptations need to be made to the existing integration to use the service fully. It's only been live since Nov 2023, and site specific forecasting is still regarded as beta.
I haven't looked too deeply at datapoint-python to see if it can be modified to allow use of DataHub, but it does look like it's gone stale given the low activity and the error above. An open issue concerning versioneer looks to be the problem. [Edit]@EJEP seems to be aware as of 3 hours ago (sorry forgot to tab refresh)]
@MrHarcombe Would a rewrite of the integration using the MetOffice examples be appropriate here, or should datapoint-python be modified?
Met Office - No Module Named 'datapoint'
Stopped working here too, after update HA
I've created a PR for that should, if my understanding is correct, hopefully resolve the issue with versioneer that has been acknowledged by the code owner for possible merging or at least to prompt resolution.
I think for longevity's sake however it would be worth pursuing a more modern API wrapper particularly as the old format is due to be deprecated. Happy to help where I can, I'd like to keep the integration in HA.
Hey, I'd sort of forgotten that the datapoint-python API wrapper was being used here. The latest version (0.9.9) should work in python 3.12 now.
I'm going to have a poke at the new Met Office API this weekend to see how much work it would be to use it with the existing code in datapoint-python. Depending on how different the new API is it might end up being easier to write a new integration from scratch though.
Thanks @EJEP that's great news, I was going to do the same to see what it would involve but you're much more familiar with your code than I am. Still happy to assist but doubt there's much more I can offer!
It would be nice if the integration could bundle in its own internal requests to the API and less on 3rd party libraries, not that they aren't effective but just so we aren't leaning on someone else if they have other things going on and can't dedicate the time. Appreciate that would involve a bit of work though.
How do we go about a request for re-enabling the Met Office integration now in HA? Or is it being "disabled" more that it just wouldn't run (but now will). Just whether I'm safe to install 2024.2 yet! EDIT: For anyone else, it stays "not-loaded" with no option to re-enable so I have skipped 2024.2.1 for now.
Problem is, integrations are required to rely on libraries that handle the API calls so that it's a proven way through and isn't relying on just some internal code.
On Sat, 10 Feb 2024, 13:28 DustyArmstrong, @.***> wrote:
Thanks @EJEP https://github.com/EJEP that's great news, I was going to do the same to see what it would involve but you're much more familiar with your code than I am. Still happy to assist but doubt there's much more I can offer!
It would be nice if the integration could bundle in its own internal requests to the API and less on 3rd party libraries, not that they aren't effective but just so we aren't leaning on someone else if they have other things going on and can't dedicate the time. Appreciate that would involve a bit of work though.
How do we go about a request for re-enabling the Met Office integration now in HA? Or is it being "disabled" more that it just wouldn't run (but now will). Just whether I'm safe to install 2024.2 yet!
— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/109301#issuecomment-1937007661, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6AZAJPKAOF7JJKFNU6LYS5YWHAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXGAYDONRWGE . You are receiving this because you were mentioned.Message ID: @.***>
Fair enough, I wasn't aware that was a requirement but good to know.
Also, is the integration open for contribution with regards to its other aspects? There are some more features that I'd like to have (pulling out templow/night low) but I'm unsure on the status of development/contributing.
I've got an integration with the new DataHub API working on a feature branch (migrate_to_datahub) of the existing datapoint-python repository. There are user-facing changes in the package API that will require changes in HomeAssistant. I had a quick look at the integration code but it wasn't obvious to me how the data from the API is used.
It might be up to a few weeks before I release an update with these changes in as I still need to document the changes and update the rest of the documentation, tests and build infrastructure.
Thanks EJEP, massive appreciation for that. I have a local dev copy of the integration now so looking forward to tinkering with the changes for DataHub when it's ready!
First I'm working on this but looks like the integration basically links Home Assistant components (e.g. ATTR_FORECAST_TEMPERATURE) with timestep elements (e.g. timestep.temperature.value) to create relevant entities through weather.py. I guess it will just depend on the changes you've made to the timesteps primarily.
Thanks for your work on this - I do appreciate it (especially as it's unlikely I'd be able to get near this until the summer holidays - I teach, and we're approaching coursework deadlines and then exam season.
It's not always easy to find the right parts of the (huge, sprawling) documentation that applies to you - I know, because I still stumble across parts. But in this case, you might want to take a look at this part https://developers.home-assistant.io/docs/api_lib_index?_highlight=pypi#trying-your-library-inside-home-assistant of the developers' documentation pages. It explains about how the goal is to have no protocol code inside an integration.
Ian Harcombe [image: https://]about.me/mrharcombe https://about.me/mrharcombe?utm_medium=email_sig
On Sat, 10 Feb 2024 at 17:57, Emlyn Price @.***> wrote:
I've got an integration with the new DataHub API working on a feature branch (migrate_to_datahub) of the existing datapoint-python repository. There are user-facing changes in the package API that will require changes in HomeAssistant. I had a quick look at the integration code but it wasn't obvious to me how the data from the API is used.
It might be up to a few weeks before I release an update with these changes in as I still need to document the changes and update the rest of the documentation, tests and build infrastructure.
— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/109301#issuecomment-1937083015, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6A5SE6AI4VOUCDBSEJLYS6YKNAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXGA4DGMBRGU . You are receiving this because you were mentioned.Message ID: @.***>
No problem and understandable, as I say I use the Met Office integration daily, it's the only weather platform I've found in HA that's actually accurate for my area. As accurate as any weather forecast can truly be, at least. I'm happy to contribute if it helps and keeps things going, it's been fun (and a learning experience).
Hi, I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.
Source: https://www.metoffice.gov.uk/services/data/datapoint
That's a little more breathing space, at least. But the other issue has been that the libraries for datapoint weren't supporting Python 3.12 - so there's been pressure from two fronts!
Thanks to all that have worked on getting this going again 🤩
Ian Harcombe
On Mon, 12 Feb 2024, 22:18 Hacs2023, @.***> wrote:
Hi, I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.
Source: https://www.metoffice.gov.uk/services/data/datapoint
— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/109301#issuecomment-1939701676, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6A3C25FZMP6OOXHZ6GLYTKILZAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZG4YDCNRXGY . You are receiving this because you were mentioned.Message ID: @.***>
Hi, I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.
Good for a little wiggle room! I suppose the main concern will be if new users can no longer sign up for the legacy Datapoint API (though the registration still appears to be open(?), haven't tried though). That in itself is a good driver to migrate to Datahub when able.
I want to have a mess around with Datapoint/hub over the weekend with the alt branch to see how that works and what might change or even need to change for the integration to migrate to it (I also really want to see about adding Night Minimum "Nm"/temp low reading). If EJEP is able and happy to finalise the Datahub branch over the next few weeks we can start working on migrating proper then!
I had definitely got muddled up between a migration of endpoints in DataHub and the retirement of Datapoint! Good that there is more breathing room.
That said, I'm still planning to press on with the migration. I plan to write some docs for what is changing in the migration, but the summary is that the custom objects are mostly replaced with dicts, and the data is no longer split up into 'days'. The main change I foresee to the current HomeAssistant integration is that information is accessed by dict keys rather than object elements.
Hi, I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.
Source: https://www.metoffice.gov.uk/services/data/datapoint
The confusion seems to have come from the banner on this page which then links to the cutoff date of April 2024 which concerns DataHub API key migration only.
I have also confirmed that registration for DataPoint is still working by creating a new account, which appears to indicate that March 2025 does indeed give more wriggle room.
Thanks for flagging this @Squelch - Like @EJEP I got muddled between them too - I think I've somehow managed to endup in the datahub emails after registering for a datapoint account, and got the associated migration emails (and then the DataHub API migration too), so got confused by the deprecation dates. I have got my API token working again from the datapoint account.
@EJEP, happy to have a look at anything/help out as I have some time on my hands at the moment!
@djpugh Easily done apparently. I'm sure we're not the only ones to have been confused by the similar naming and dates...
Are the 3 hourly entities no longer functioning? Mine are all unavailable. Only the daily weather (with a menu to see the hourly) are available.
Assuming you've just updated to 2024.4, the daily and hourly forecast entities have been merged: https://www.home-assistant.io/blog/2024/04/03/release-20244/#backward-incompatible-changes
Assuming you've just updated to 2024.4, the daily and hourly forecast entities have been merged: https://www.home-assistant.io/blog/2024/04/03/release-20244/#backward-incompatible-changes
I used met office 3 hourly forecasts and it was working untilt he lastest upgrade.
met office 3 hourly was the entity.
Is there any way to show the hourly forecast in the main card? That would be really useful.
Is there any way to show the hourly forecast in the main card? That would be really useful.
Second this.
Also: I was referencing the attributes of various 3 hourly forecasts, including temperature, humidity, precipitation probability in a number of templates to drive a number of functions and actions in my instance, which are now unusable. The data is nolonger shown /shown correctly under attributes. The 3 hourly entities now just show one value, rather than multiple 3 hour time periods listed under attributes.
AFAIK the various entities are all still available - I was using the 3-hourly weather in an AppDaemon script and had to change the naming, but it still seems to be working.
I think I need to put through a docs change to reflect the changes that were made in 2024.4.1 to bring the daily and 3-hourly under one entity...
If you look in the settings, search in entities for "met office" you should still find the same default set of entities listed although some of the names may have slightly changed; for me, the 3-hourly weather is called "sensor.market_harborough_weather" because of my location. I also have similar named enabled entities for _probability_of_precipitation, _uv_index and _wind_speed. If you then check your filters and include the disabled ones you can see where the others are.
The daily sensors, however, are all banned slightly differently, as my default is showing as "weather.met_office_market_harborough_daily".
But, for me, everything still seems to be there.
Hope that helps?
Ian Harcombe
On Sat, 6 Apr 2024, 01:50 systanford, @.***> wrote:
Is there any way to show the hourly forecast in the main card? That would be really useful.
Second this. I was referencing the attributes of various 3 hourly forecasts in a number of templates to drive a number of functions and actions in my instance, which are now unusable.
— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/109301#issuecomment-2040833705, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6A5SPWWTPMIYGS7TQK3Y35BGTAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQHAZTGNZQGU . You are receiving this because you were mentioned.Message ID: @.***>