open-meteo
[Suggestion] API for KNMI
New updated KNMI weather models: https://dataplatform.knmi.nl/group/weather-forecast
Beware that KNMI will likely update their domains and grid sizes this year.
Just wanted to add my bump of support for this; would be great to have an easier way to get KNMI forecast points than downloading the whole grid via their site.
I will probably start adding new models in a couple of weeks. Currently, I am occupied with expanding server capacity and cleanup all current model integration...
New HARMONIE model might be finally released :)
We are excited to inform you, that a new version of our weather model HARMONIE will be released in a betaversion on our open data portal March 20th 2024.The release of our new version is a part of DMI’s migration to our new super computer.
You can find all information regarding the migration and changes in relation to the new version of HARMONIE on our technical documentation site. We will continuously update this page with more information.
Transition period
The technical documentation for the new versions will be available from the release date, and there will be a transition period, where both old and new versions will run in parallel. The transition period will announced on our technical documentation site as soon as possible. A migration guide will also be released on the site.
What does this entail for you as a user?
The new version of HARMONIE will come will improve horizontal and vertical resolution. Our verification of the new version shows that the new version scores better on a variety of parameters such as precipitation, wind, and wind gusts.
There are, however, a number of changes, which will occur, when we change to the new HARMONIE versions.
Our HARMONIE models will change names with the new versions. In the table below, the name changes are shown:
The new HARMONIE versions will also differ from the old versions in a number of ways, which you can read more about on our technical documentation site.
Wave Model (WAM) and Storm Surge Model (DKSS)
The new HARMONIE model is a part of DMI’s migration to a new super computer. Our wave model (WAM) and storm surge model (DKSS) will also be migrated to the new super computer in the coming months. There will, however, not be made any changes to our WAM and DKSS in the migration, and migration is planned to go seamlessly, so you should not notice when we migrate.
We are excited to inform you, that a new version of our weather model HARMONIE will be released in a betaversion on our [open data portal](https://opendatadocs.dmi.govcloud.dk/DMIOpenData) March 20th 2024.The release of our new version is a part of DMI’s migration to our new super computer.You can find all information regarding the migration and changes in relation to the new version of HARMONIE on our technical documentation site. We will continuously update this page with more information.
Transition period The technical documentation for the new versions will be available from the release date, and there will be a transition period, where both old and new versions will run in parallel. The transition period will announced on our technical documentation site as soon as possible. A migration guide will also be released on the site.
What does this entail for you as a user? The new version of HARMONIE will come will improve horizontal and vertical resolution. Our verification of the new version shows that the new version scores better on a variety of parameters such as precipitation, wind, and wind gusts.
There are, however, a number of changes, which will occur, when we change to the new HARMONIE versions.
Our HARMONIE models will change names with the new versions. In the table below, the name changes are shown:
Old HARMONIE names New HARMONIE names HARMONIE NEA HARMONIE DINI HARMONIE IGB HARMONIE IG
The new HARMONIE versions will also differ from the old versions in a number of ways, which you can read more about on our technical documentation site.
Wave Model (WAM) and Storm Surge Model (DKSS) The new HARMONIE model is a part of DMI’s migration to a new super computer. Our wave model (WAM) and storm surge model (DKSS) will also be migrated to the new super computer in the coming months. There will, however, not be made any changes to our WAM and DKSS in the migration, and migration is planned to go seamlessly, so you should not notice when we migrate.
Exciting! Are we expecting a higher resolution HARMONIE-AROME from KNMI too? Or has that happened already? I can't see any announcements but note @phillipseamore's comment about changing grid sizes
I will check the open-data server from KNMI and DMI after the upgrade. I am also not quite sure who is running which HARMONIE model/domain.
KNMI new UWC expected in April, test files available. https://www.knmidata.nl/actueel/nieuwsbrieven/open-data-nieuwsbrief/2024/open-data-februari-2024
The new models from DMI are not public just yet but can be downloaded by changing the model name, e.g. "harmonie_dini_sf" instead of "harmonie_nea_sf"
Just wanted to update you all that much of the model output is available now, including the 6 member hourly updated ensemble that covers a large part of Europe.
https://dataplatform.knmi.nl/group/weather-forecast?q=UWC&sort=metadata_modified+desc
Had a brief look and it looks feasible to integrate. I will need some additional components to work efficiently with tar files (I want to stream everything in memory :D).
I noticed additional GRIB tables are required. I found some here. Do @phillipseamore do you know if there is an official source for KNMI grib tables?
In case it helps, I had a look around when scoping this up and the only people who seem to have full GRIB tables are people who have emailed KNMI for them. That's where the .zip in what you linked came from; not sure if it works with the new UWC GRIBs too.
@patrick-zippenfenig No I don't have a source, however the unkowns seem to correlate with Harmonie parameter ID's (e.g. : https://hirlam.github.io/HarmonieSystemDocumentation/dev/ForecastModel/Outputlist/
If the tables you linked don't contain all of them the DMI definitions might cover them: https://opendatadocs.dmi.govcloud.dk/dmi_grib_definitions_v1.0.0.tar
Seems to work. I will check tomorrow if all fields can be mapped. Working with the API is also a bit annoying due to Auth and two step URL generation. On the other hand, it's based on AWS S3 and quite fast.
You can just barely see the alps and Italy
Making progress.... Interestingly the Netherlands nest produces different forecasts https://english.knmidata.nl/latest/newsletters/open-data-newsletter/2024/open-data-march-2024
There are still a couple of issues to solve. E.g terrain elevation data is missing, derived variables need to be defined, etc
EDIT: Never mind the differences. There was a bug in the grid projection
A preview version is available: https://staging.open-meteo.com/en/docs#latitude=52.374&longitude=4.8897&hourly=temperature_2m&models=ecmwf_aifs025,knmi_harmonie_arome_europe,knmi_harmonie_arome_netherlands
EU and Netherlands domains are available. Only deterministic model. Updated every hour (if the update routines actually work ;-)).
List of downloaded variables here: https://github.com/open-meteo/open-meteo/blob/603ce3cb38e1d7fc81393b99c0dfb7ddc30649eb/Sources/App/Knmi/KnmiVariable.swift#L6. Note: wind on model levels 100/200/300 is only available for Netherlands nest.
It's still a bit of work and I have to contact KNMI for some details (gpm/z at surface is missing)
Brilliant - such quick prototyping! For the wind heights, I can't see any checkboxes to test them in staging just yet
Super!
Is KNMI version of DINI distributed at 676x564 instead of the full resolution 1906x1606 like at DMI? If they had the same resolution all the constants from DMI's DINI should be the same.
As far as I know the NL domain should just be a simple cutout but you might get different results because of these resolutions differences.
Hopefully someone at KNMI can provide some insight into this.
With the ensemble, in production it's used lagging over 6h so it ends up as 30 members total. I guess if anyone wants to use that in the same way the historical forecast API could be used?
For the wind heights, I can't see any checkboxes to test them in staging just yet
Yeah, that's a general issue. Each weather model offers different weather variables, heights and pressure levels. I have not yet found a good solution to build a UI for that.
Is KNMI version of DINI distributed at 676x564 instead of the full resolution 1906x1606 like at DMI? If they had the same resolution all the constants from DMI's DINI should be the same.
KNMI is using a rotated latlon projection with 676x564 and updates every hour. I also had a look at DMI and the full resolution is very tempting, but only 3-hourly updates. Projection is Lambert conformal, but wind direction has not been adjusted to projection. I am getting a bit confused why they use different projections, standards, etc....
With 36 GB per run, it is also more data. I am also upgrading all API servers right now, so it is feasible to store all this data.
With the ensemble, in production it's used lagging over 6h so it ends up as 30 members total. I guess if anyone wants to use that in the same way the historical forecast API could be used?
It should be 15 members in total with 5 members per run.
DMI Harmonie AROME is now integrated for testing as well:
- 2 km resolution, Lambert CC projection (KNMI ~5.5 km Rotated latlon projection)
- ~36GB per run for surface variables, download time roughly 18 minutes with 1:30 minutes processing time (16 cores)
- 1 hourly values, 60 forecast hours
- updates every 3 hours, KNMI has 1-hourly updates
- More variables compared to KNMI
- Preview here (you might have to hard reload / empty browser cache in case DMI is not visible)
Technical notes:
- DMI uses different GRIB tables. Data can be read without additional tables, but has some wrong definitions (e.g. wrong units for solar radiation,
dswrfcontains direct radiation instead of downwards radiation. Using the DMI GRIB tables as linked above makes no difference. Even official DMI documentation listsunknownparameters. - The inclusion of direct radiation in DMI is highly appreciated. Interestingly the solar separation model seems to work quite well to estimate direct radiation for KNMI. DMI also contains DNI (direct normalised irradiance), but there is basically no difference compared to calculating it using solar position algorithms (I compared results which are almost identical).
- Wind U/V components must be corrected for projection. Without this correction step, wind directions are 15-20° wrong. I am pretty sure that not all GRIB users will notice (or even care) that wind speeds not correct
Currently, I am decoding the following DMI parameters:
https://github.com/open-meteo/open-meteo/blob/42445e9834552591431e605a5e30bd0830713c82/Sources/App/Dmi/DmiVariable.swift#L6-L51
https://github.com/open-meteo/open-meteo/blob/42445e9834552591431e605a5e30bd0830713c82/Sources/App/Dmi/DmiDomain.swift#L12-L35
ToDo:
- Elevation and land/sea mask for KNMI domains. KNMI harmonie Europe does not contains any gph/z at surface height
- Cloud cover total seems buggy in DMI (low, mid, high is fine). Probably I misinterpreted it and it is some kind of fog (GRIB code
ccat 2m above ground) - Figure out a way to convert
snow_depth_water_equivalenttosnow_depthin cm (unfortunately there is no snow density...) - Validate all parameters
- Build a UI that works better with various wind levels like 50,100,150,200,250,300 metres
- Consider
dmi_seamlessorknmi_seamlessdomains with IFS0.25 combination
This going fast.
There are a few better definition in this list: https://opendatadocs.dmi.govcloud.dk/Data/Forecast_Data_Weather_Model_HARMONIE_DINI_EDR
DMI total cloud cover (0m) is 0.0-1.0 not 0-100 and the 2m level appears to be fog (and would be nice to have as well as it has performed very well with the better DINI resolution).
0-17-193 is probability of lightning 0.0-1.0
0-1-192 is categorized precipitation type at surface: 0 drizzle 1 rain 2 sleet 3 snow 4 freezing drizzle 5 freezing rain 6 graupel 7 hail
0-6-199 is fraction of visible light that goes through the cloud layer 0.0-1.0 (perhaps better than tcc if inverted?)
The previous hour min/max 2m temp would be interesting as well. Could it be that the T2M is staging now is actually the previous max and not instant, from a little checking it seems to correlate better with observed hourly max today in DK than avg?
Loving this, will be a great addition!
Previews for both DMI and KNMI endpoints are now available:
- https://dmi-knmi.open-meteo-website.pages.dev/en/docs/dmi-api
- https://dmi-knmi.open-meteo-website.pages.dev/en/docs/knmi-api
Cloud cover at 0m above ground seems to be total cloud cover. Cloud cover 2m is now integrated and seems to work surprisingly well for fog. I compared some locations in Iceland and it seems reasonable.
Cloud base and top will be included with the next downloader run. Always looks nice as a plot :D
I also added dmi_seamless and knmi_seamless models which are combined with ECMWF IFS 0.25° to produce a decent 10 day forecast. Most likely I will also include those models in the best_match selection.
ToDo:
- Elevation and land/sea mask for KNMI domains. KNMI harmonie Europe does not contains any gph/z at surface height.
- Figure out a way to convert
snow_depth_water_equivalenttosnow_depthin cm (unfortunately there is no snow density...) - Include DMI / KNMI in best match
