hass-weatherflow2mqtt
hass-weatherflow2mqtt copied to clipboard
briis
Weather Condition generating locally
The below are the only inputs allowed for HA weather condition. Currently this is pulled from the web API. I was toying with the idea of expanding my weather condition template to pull this locally. I understand I can't get the forecast local only; but I like as much self-hosted as possible. I think the only things I can't verify is clear-night, due to light conditions at night. Thoughts below: ‘clear-night’ can't check if fog at night since no light level to compare to (open to ideas) ‘cloudy’ can get a base line lux amount and compare and if not any other precipitation ‘fog’ same as cloudy but with window below a determined temperature ‘hail’ sensor for this ‘lightning’ sensor for this (maybe any number in last 3 hours) ‘lightning-rainy’ combine sensors for this ‘partlycloudy’ same as cloudy, with maybe a trend/derivative sensor to determine intermittent light levels ‘pouring’ above a certain rain rate ‘rainy’ sensor for this ‘snowy’ a little more complicated but maybe temperature combined with blocked UV sensor and rain sensor ‘snowy-rainy’ same as snowy but easier to assume around 0C / 32F ‘sunny’ easiest just base on UV/lux level ‘windy’ wind above a certain speed or gust above a certain speed ‘windy-variant’ not sure what this is checking ‘exceptional’ dewpoint between 50-60F, temperature between 68-77F, UV below 2.5 index, not any of the above (except sunny).
So...the actual point of this... should I make my condition template and then PR to the readme example or should we incorporate this to give another calculated sensor output? I'm leaning toward a sensor, but this would add a lot more code and wanted to run it past you before a PR without a big change with any other info. I would keep the current web api returned status as you have it and create some other named sensor. Thoughts?
This is high on my list as well, and thinking through how I might do the same locally. I have already expanded my use of current conditions/icon selection in HA with a template sensor and pulling information from multiple sources, just not to the extent as you contemplate above... yet. I suppose the HA weather component is necessarily limited to a LCD of conditions. To me, partly cloudy, mostly cloudy, and cloudy are three very distinct and recognizable conditions. And I was hoping to be able to distinguish between them with logic as you mention. I am also not a huge fan of how WeatherFlow does its current condition. They will use something like "rain possible" even though the sky is clear or partly cloudy. To me, current condition is what it is NOW, the possibility of rain is a forecast. Dark sky does a better job of this, so for now I am using their current conditions to drive much of my NOW, sensor, at least until apple shuts it down next year.
I am certainly happy to contribute to this however I can. I think it could be a significant value-add. But I dont think the calculations will be trivial to make it work for everyone. Local latitude, current season, and sun angle/time of day will all likely need to be factored in. So, it may be more feasible as a suggested template sensor with perhaps some guidance on tweaking for local use. In any case, count me in.
What spurred it for me was exactly what you said above, I had a condition of raining as reported by api but there was no rain. I would like it to be actual not predicted just at you stated. I will am open to any ideas on accomplishing any of the determinations. I think I have a handle on most of them but, as stated, I don't know how to handle some conditions at night mainly clear/cloudy/foggy. I might be able to use dewpoint and temperature for a fog determination. I do have some other sensors at my disposal but I want to try to do this just thru my Tempest so it will benefit others also. I have a PurpleAir air quality sensor that determines particulate matter in the air; I could use it to determine between smoke and clouds but that is trivial and I'm ok with cloudy vs smoky; especially since there is no smoke in weather card.
On a slightly separate note you could look up purpleair on the HA forums to see some of the gauges I've done. If you look up pollen, I have an updated gauge with a color coordinated background to describe conditions such as cool for temperature or dry for dew point.
I will see what I can knock out this weekend. I am open to ideas either now or after I post my first attempt to see if anything can be smoothed over.
You know, before your post, I was blissfully unaware of the Purlple Air sensor. Now I NEED one! My wife is not going to be happy with you ;) Good to hear I am not the only one that feels that way. I will post something on Monday on the Weatherflow forum re: the current conditions. Their AI should be able to do a great job with satellite and radar data if they were so inclined. Dark sky has the best model for my area, and they are right most of the time. But they may not be around for ever. And like you said, it would be best if it could be done all local.
Hi both, I believe this is an interesting topic, but as you both point out, it can be tricky to get all the conditions right, based on the data we have local access to.
I guess the best way to start is by what Glenn has already done, trying to make some algorithms for each condition, and then tune these a as we go a long. I will be happy to make a new sensor, we could call it local_condition and then start putting in the math.
If everything fails, what would be the default condition: Cloudy?
I think it is good that you posted on the forum Glenn, there might be others already doing some of this.
As soon as I have time, I will start building it, based on what is already here.
@chasut enjoy:
I am still tweeking the backgounds on Dewpoint and Temperature that is why there are no words there yet. The UV background started it all, I saw someones background for their UV index and I modified it for other things. I was using an integration that pulled the UV data before, but I removed that integration since I have the Tempest now. So from this the UV is the only thing coming from the Tempest, I still am using the PurpleAir for Temperature/Dewpoint/Humidity and of course Air Quality. The Tempest and PurpleAir track fairly well except for humidity; the Tempest seems to run high sometimes like 99 and 100% too often.
Here is my other dash (AQI is duplicated here)
Unfortunately for me the allergy/asthma/cold&flu is pulled from an cloud API. There are local options for all but the Cold&Flu, but they are a little pricey; the allergy monitor I looked at was $8000. I have most of the Asthma already with my PurpleAir since 2.5PM is a big part of it; I have not priced ozone detectors yet.
The PurpleAir has a local web server on the device that allows you to look at the device directly, or you can look at the web map, or you can pull the data from the Rest API (local and/or web) (which is what I do). I got the PurpleAir after wildfires last year and I do not regret that purchase at all. My wife has asthma and that little bit of information helps her out a lot.
I might do this, but it will not be prioritized right now. But please keep posting how you would calculate the different Conditions, then I will start adding when I have time.
I'm not sure what happened, but I had updated a lot of the conditions in the above post...I probably closed the tab without saving the edit.
Establishing order of checks (priority): ‘lightning-rainy’ 10 < (sensor.wf_lightning_count_3_hours < 450 (attempt to eliminate false reports from noise on detector) & sensor.wf_precipitation_type > 0
‘lightning’ 3 hour lightning count 10 < (sensor.wf_lightning_count_3_hours < 450 (attempt to eliminate false reports from noise on detector) & sensor.wf_precipitation_type > 0
‘hail’ sensor.wf_precipitation_type = 2
‘snowy-rainy’ sensor.wf_precipitation_type > 0, sensor.wf_temperature = 32F / 0C
‘snowy’ --- snowy is a guess since there is no 'sensor' that detects snow (temperature in C) sensor.wf_temperature < 1.44 - 0.08 * sensor.wf_humidity) (temperature in F) sensor.wf_temperature < 34.29 - 0.14 * sensor.wf_humidity) and sensor.wf_temperature < 40F / 4.44C and sensor.wf_illuminance < 1000 if sun angle > 0
‘pouring’ sensor.wf_rain_rate > 0.31 in/hr (7.8 mm/hr) or sensor.wf_precipitation_type = 3
‘rainy’ sensor.wf_precipitation_type = 1
‘windy-variant’ Wind Speed > 25mph (11.17 m/s) & cloudy
‘windy’ Wind Speed > 25mph (11.17 m/s)
‘fog’ Fog is likely when: Air Temperature - Dew Point < 4.5F (2.5C) Relative Humidity > 95% Wind Speed (above) > 5 knots / 5.7 mph / 2.57 m/s Wind Speed (below) < 10 knots / 11.5 mph / 5.14 m/s Clear skies overnight (???) maybe from Pressure '> -40F (typically >14F) *** near seawater >70% Relative Humidity and wind speed <10 knots (5.14 m/s)could cause fog (might skip this caveat for now) Need more conditions for fog, it looks like this might throw a lot of false fog. There is some trends for heating/cooling but most would not have a derivative sensor for air temperature. Cooler air temp than ground temp, determine if daily temp max (24hr) is greater than current temp (more to come) --Use visibility calculation as condition
‘cloudy’ reduced luminescence with a steady tread
‘partlycloudy’ trend of alternating lower and higher lux
‘clear-night’ sun angle < 0 and no other events
‘sunny’ no other events and sun angle > 0 (minimum of 10,000 lux)
‘exceptional --?? (looked at some weather cards and does not seem to be used)
Need to look at more data and adjust for latitude and time of day. 0.0001 lux – Moonless, overcast night sky. 0.002 lux – Moonless clear night sky with airglow. 0.27–1.0 lux – Full moon on a clear night. 3.4 lux – Dark limit of civil twilight under a clear sky. 100 lux – Very dark overcast day. 400 lux – Sunrise or sunset on a clear day. 1000 lux – Overcast day 10000–25000 lux – Full daylight (not direct sun). 32000–100000 lux – Direct sunlight.
Still gathering information. Discussed with an individual on the WeatherFlow forums that calculates cloudy based on UV level comparison with historical data for each month for a given latitude; I could go this route but this complicates it much more than I anticipated. I am still looking for a 'simple' formula for UV or even solar radiation for latitude. I might end up breaking it up to latitude segments of 5-10 degrees for an approximation and then see how to figure the bell curves for daylight for a given time of year for a given latitude.
I will test what I have as a sensor in Home Assistant and fine tune as needed. Once I have it fairly reasonably working for me, then I'll put it on the forum for HA and hopefully get others at different latitudes to test.
Working on 'normal' UV levels. I did not use the standard symbols for below, just a description. This is just for documentation at this time, nothing to be implemented yet until I get further along.
Need the following:
Solar Declination Angle = 23.45 * sin(360/365(d-81)) d is Julian day (ie January 1st is "1")
Max Elevation at Solar Noon = 90 + h * (Latitude - Declination Angle) h is +1 for Northern Hemisphere, -1 for Southern Hemisphere (imputed variables like elevation) If solar elevation is >90 then subtract from 180
Elevation (already available) for each 300 meters in elevation above sea level, UV levels increase by 4%
Assumption that weather station UV sensor sees direct sunlight most of the day.
More calculations to come.
I will start looking at this, in a few weeks when my vacation is coming up, and see what I can do.
I think I have to reevaluate how I am determining cloudy. Today was completely overcast but UV levels were up significantly. I know that clouds can reflect some UV back down but I saw UV index of 6 without clouds (two days ago) and 11.76 with clouds (today). Solar Radiation and Light Levels were both down though (yes, I know that UV is a different sensor).
I might go with comparing current light levels to calculated light levels (based on time of day, latitude, and time of year); and also a comparison of current UV to current LUX.
I think I can get away with only calculating for the Northern Hemisphere (NH) and then make a 180 day adjustment for the Southern Hemisphere (SH). The one thing I can not get locally is Ozone levels which affect UV levels, but they are fairly constant during the SH summer/ NH winter Jan/Feb/Mar.
This has become a much deeper rabbit hole than I expected and I'm still digging down. I am almost to the point of making my predicted UV/Lux sensor in HA so I can compare to actual and also clouds.
It is very easy to look at either the UV or Lux or Sol Rad graphs and see when there are clouds. This could be used if the day started out sunny and cloud(s) moved in causing a more rapid than drop than sunset; but if full clouds for the whole day then there would not be any drops.
Just documenting my thoughts.
@briis is there a easy way to determine if the forecast connection is down? I know there is a similar issue here but my need is to provide the local conditions (mainly the current conditions icon) only when there is no internet. Yesterday I was without grid power and internet at home for 7 hours and I had all my WF sensors but I found that I really missed my big animated icon of the current weather conditions. So I was thinking of supplying my estimations of current conditions when either there is no connection to the weather flow server or someone has opted to not get forecast data. I will not be able to cover all conditions but I can get it pretty close, at least for my location. I might just start with the easy status like rain (day), rain (night), sunny (day), clear (night). I won't be able determine clouds at night and clouds in day still require some work on my end but the previous is easy then I can start adding from there.
@GlennGoddard There are couple of ways to test if the connection fails and/or the data we have for forecast is empty, and then based on that switch to the local format. I guess the most important part is that the dataset is formatted in the same way as the data we retrieve from WeatherFlow. I have attached a sample json file - this is what gets returned when calling the forecast. I can help with generating that, if you make the dataset and then implement the switch-over function.
I have not abandoned this, I have been very very very busy lately; hopefully I will have some time over the Christmas break to work on it.
A forecast in python that might be useful https://github.com/jim-easterbrook/pywws/blob/31519ade415545e9cd711237b98aad33d070c1f9/src/pywws/forecast.py
http://www.beteljuice.co.uk/zambretti/forecast.html
Snow Line https://www.bordersweather.co.uk/wxsnow.php?1638214717#data
The Zambretti Forecast is the same that my Davis Vantage station delivers. Honestly I have never checked how accurate that is, but I assume it is better than nothing, if without Internet connection.
It would be great to the weather condition locally! I also reaslied a couple of days ago that Tempest Weather Sensor showed rainy despite there was no rain.
The forecast, provided by the government with local weather stations everywhere, stated partlycloudy instead, which was correct. Luckily, there is an integration for that provided forecast for HA (any other forecast such as SKY or whatsoever will work too).
You can take first your local weather forecast and then tempest, or, even better, first your local weather forecast, then your "country-forecast" which is probably better then temptest, and then, tempest.
I can't really contribute on this feature of full local forecast, but for now, I created a sensor which merges the weather condition in the following priority:
- take the local Tempest forecast which is rain, pouring, hail, lightning, lightning-rainy and windy - this works offline
Since we can't calculate anything which involves a "mixture" where we only have one of the two conditions (rain and wind), or if it is exceptional (in particular: snowy-rainy, windy-variant and exceptional), we need to check the internet forecast first. For example, we need first to check if its snowy-rainy (Internet) and just after that, if its rainy (Local), otherwise we will be having a wrong state (rainy instead of snowy-rainy.
You may adapt the parameters to your needs (lightning/lightning-rainy: strike <=1km away within the last 10mins, windy: 32-48km/h). I created a km/h sensor to convert the m/s that is provided by the integration. If you have the m/s sensor only, then divide 32-48km/h by 3.6 and you're good to go
-
if the above is not true, then take the condition from the "country-weather-forecast" (relies on an internet connection).
-
if the above both is not true, then take the Tempest online weather forecast (relies on an internet connection).
You can then use this sensor to create a weather entity which has, at the time of writing, the most possible local weather condition :)
Here is the code for your configuration.yaml, you may need to adapt to your local translation (currently, 4 languages are supported by the integration. XYZ is the entity of your "other" online forecast:
# Sensor Weather state merged (Local, XYZ, Weatherflow)
- platform: template
sensors:
weather_state_merged:
value_template: >
{% if states('weather.xyz') == 'exceptional' %}
{{ states('weather.xyz') }}
{% elif states('sensor.weather_home_lightning_distance') <= '1' and (as_timestamp(now()) - as_timestamp(states('sensor.weather_home_last_lightning_strike'))) < 600 and (states('sensor.weather_home_precipitation_type') == 'Rain' or states('sensor.weather_home_precipitation_type') == 'Heavy Rain')%}
{{ 'lightning-rainy' }}
{% elif states('sensor.weather_home_lightning_distance') <= '1' and (as_timestamp(now()) - as_timestamp(states('sensor.weather_home_last_lightning_strike'))) < 600 %}
{{ 'lightning' }}
{% elif states('weather.xyz') == 'snowy-rainy' %}
{{ states('weather.xyz') }}
{% elif states('sensor.weather_home_precipitation_type') == 'Hail' %}
{{ 'hail' }}
{% elif states('sensor.weather_home_rain_rate') | float >= float(10) %}
{{ 'pouring' }}
{% elif states('sensor.weather_home_precipitation_type') == 'Rain' %}
{{ 'rainy' }}
{% elif states('weather.xyz') == 'windy-variant' %}
{{ states('weather.xyz') }}
{% elif float(32) <= states('sensor.weather_home_wind_speed_avg_kmh') | float <= float(48) %}
{{ 'windy' }}
{% elif not states('weather.xyz') == 'unavailable' or not states('weather.xyz') == 'unknown' %}
{{ states('weather.xyz') }}
{% else %}
{{ states('sensor.tempest_hub_weather') }}
{% endif %}
friendly_name: Weather home State Merged
icon_template: mdi:weather-partly-cloudy
@badewanne1234 this may not be a bad approach to compare local station data to WeatherFlow's current condition when there is internet and the forecast option is setup. I was working off the idea of providing the current conditions completely locally, to allow for no internet; but I like your idea of comparing local and web data for the current condition.
@GlennGoddard Well its not really "comparing", its preferring local weather condition if available, else take the online one. Of course I'd rather prefer to have all states to be taken local, but as stated, that goes byond my capabilities :)
So for now, I'm sticking with the above approach, and for me it works very well, the current weather condition is much more accurate now. I'm also updating the above post, I'm still finetuning the weather state entity. It will be very interesting to see how the lightning calculation works, I probably need some tweaks there, but for that I'll need data from a lightning event... I have additional timers setup (not includeded in the above post) to not have "cloudy" as soon as it stops raining, I wait a little while before switching from "rainy" to "cloudy" for example.
Narrowing in on Cloudy (will need a more data points at solar noon on a sunny day). Overview: compare measured solar radiation to calculated Still work in progress
The higher the solar elevation the higher solar radiation (the assumption is made that the weather station's sensor is parallel with ground). 90 degress is max solar elevation.
Solar Elevation: α = sin^(-1)((sin(ϕ) * sin(δ)) + (cos(ϕ) * cos(δ) * cos(h))) where α is Solar Elevation δ is declination h is angle hour ϕ is latitude
δ = (-23.44) * cos(360/365 * (d + 10)) where d is number of days since January 1st (julian date) + fractional for hours
h = 15 * (LST -12) where LST is local solar time in 24 hr format
LST = LT + TC/60 where TC is Time Correction Factor LC is Local Time (24hour with fractional minute i.e. 30 mins = 0.5)
TC = 4(λ - LSTM) + EOT where λ is Longitude LSTM is Local Standard Time Meridian EOT is Equation of Time
LSTM = 15 * ΔTutc where ΔTutc is the difference of local time from UTC (also equal to the time zone) (example Los Angeles is -7)
EOT = 9.87 * sin(2*β) - 7.53 * cos(β) - 1.5 * sin(β) where β = 360/365 * (d - 81)
I think the way I will go with this is to convert this to a percentage of max solar radiation for a given time and day and compare that with the measured solar radiation. If in a certain (yet to be determined) range then it is sunny. Specifically within a range since higher than 'max' solar radiation is an indication of fluffy white clouds reflecting UV light back down to the ground (as long as not directly between sun and monitor i.e. blocking direct sun light). If outside that range and no other conditions such as rain then cloudy. This also makes the assumption that station is never in the shade (not always practical but the easiest approach). This is also only valid during 'normal' daylight hours.
Once I have a good working approximation for the Northern hemisphere, I will try to simulate for the Southern. I 'think' I have that already figured into the above calculations but I'm not entirely sure.
As much as I would love to continue to use the Greek alphabet in the code, I will use more appropriate Latin characters for variable names.
@badewanne1234 I can always use more data points to see how accurate this is; admittedly this is not ready for anyone else to try since I don't have the approximation done yet but any input is welcome.
I have considered the consideration of solar eclipses but I don't know if I want to go there. It could be done with some packages but I don't think it is worth the effort (maybe).
wow, looks like you're an expert! Let me know if I can assist with testing etc
@badewanne1234 If it is not inconvenient, could you log solar radiation at solar noon if no clouds present along with the date. As you can get the data is fine; I can (unfortunately) go months without any direct sun in the winter. Eventually I would like data with clouds but I don't want too much data (yet).
Things that would help, but not required if you don't feel comfortable: Elevation above sea level of the station Lat & Long (if more comfortable approximate location is fine)
Don't post here yet. Probably IM in either the Home Assistant or WeatherFlow community boards.
Partly Cloudy and Cloudy, I still have to figure out. I think I need a time comparison for partly cloudy conditions to see if there are any up/down changes in solar radiation during a time frame (excluding solar noon, sunrise, and sunset).
I had no idea that figuring out local conditions would be this complicated when I started this.
@badewanne1234 If it is not inconvenient, could you log solar radiation at solar noon if no clouds present along with the date. As you can get the data is fine; I can (unfortunately) go months without any direct sun in the winter. Eventually I would like data with clouds but I don't want too much data (yet).
Things that would help, but not required if you don't feel comfortable: Elevation above sea level of the station Lat & Long (if more comfortable approximate location is fine)
Don't post here yet. Probably IM in either the Home Assistant or WeatherFlow community boards.
Sure not a problem. With solar radiation, you mean illuminance, right? So ideally, I would log illuminance, sun elevation, date&time and the weather condition... Lets see if there is something available to log this automatically into a CSV file.
@badewanne1234 Actual Solar Radiation measurement; same line in the APP as UV thou. The units are W/m^2 The tempest has UV, Brightness and Solar Radiation. Brightness and Solar Radiation do use the same sensor and are representative, so maybe that will work. On second thought, luminescence should work; if you could log both
Update:
I decided to test these in Node-Red since it is easy to modify and it was very easy to pull in lat/long. I still had to establish station elevation since that is not sent to MQTT (nor does it need to be). I am currently testing 'cloudy' based on estimated Solar Radiation vs measured Solar Radiation, my unfortunate issue is that I have lots of data for cloudy days but none with any sun. I am exporting my results from Node-Red to MQTT and then using MQTT explorer to look at the output in graphical form. I am testing a few algorithms for Solar Radiation estimation and I can see that sometimes they appear too high at solar noon but I'm not sure since I have not had a sunny and non-cloudy day in weeks.
If anyone else utilizes Node-Red I would be willing to share my node for weather. I pull in the data from the Tempest via Home Home Assistant and not MQTT nor UDP directly. My intention is to get this working as much as possible before incorporating into this repo. So far we would need Latitude and Longitude added as a variable (just like elevation is already provided). I am making the assumption that the time for this container is in accurate local time (stating this in the case anyone is using an isolated machine that does not have a good time source); I think currently as long as time is within 16 mins then it will work ok. Just to note that time is important since the earth rotates 1 degree every 4 minutes, which in turn effects the received (and estimated) sunlight on the top of the weather station and therefore the Solar Radiation.
my temptest is exposed to the sun directly. I have some data from the DB, but I need to compare the timestamps from the different sensors (working on this). I might do a live logging instead (write to file every 2mins or so), it is supposed to be sunny mid next week here.