rtl_433
rtl_433 copied to clipboard
merbanan
Change all uv fields to uvi, BREAKING change to UV sensors
BREAKING change to Acurite-Atlas, Bresser-6in1, Bresser-7in1, Cotech-367959, Emax-W6, AOK-5056, Oregon-UVR128, Oregon-UV800, Vevor-7in1.
This changes all "uv" named UV Index keys to "uvi" for a proper distinction between a raw UV reading and UV Index values.
Part of #3103
I wonder what the two Bresser, the Cotech and FineOffset WS80 one decimal doubles are
DATA_FORMAT, "%.1f", DATA_DOUBLE Either some very obscure way of presenting a usually integer UV Index scale with additional in between decimals, or also some other raw data not indicating the UV Index at all?!? With the latter also requiring to rename their "UV Index" label.
The initial implementations of these decoders might have moire insight into this, or if someone has one of these stations.
Code and notes on all changed fields indicated UV Index values. Some present decimals, yes.
Ok then, not that I have ever seen or known about decimal UV Index values, but if that is the case the changes look fine :)
Thanks for the disambiguation 👍
I am 99% sure the UV index is not constrained to integers. Precision greater than 0.1 does not really seem useful, so .1 seems good to me.
I am 99% sure the UV index is not constrained to integers. Precision greater than 0.1 does not really seem useful, so .1 seems good to me.
Yes, it is not, but due to the limitations of current UV sensor technology, especially in consumer products, and usually being limited to only one UV spectrum, most stations do only present an integer value, at least the ones I have seen and used so far. With all these limitations I think that with any station showing .1 precision it is more of a marketing decision than a precision taken too seriously.
I am 99% sure the UV index is not constrained to integers. Precision greater than 0.1 does not really seem useful, so .1 seems good to me.
Yes, it is not, but due to the limitations of current UV sensor technology, especially in consumer products, and usually being limited to only one UV spectrum, most stations do only present an integer value, at least the ones I have seen and used so far. With all these limitations I think that with any station showing .1 precision it is more of a marketing decision than a precision taken too seriously.
Your comment is lacking a footnote to test results :-) There's precision and there's accuracy, and I've found that things that are repeatable even if off are still very useful to see relative changes - like 0.01C resolution in temperature. I really don't know what is justifiable (and without a proper evaluation nobody else does either). But, rtl_433's approach is, generally, to represent what's in the packet, at least if it isn't goofy. An index that more or less goes from 0 to 12 using 0.1 does not seem silly. I would agree that 0.01 is unlikely to make sense.
An index that more or less goes from 0 to 12 using 0.1 does not seem silly. I would agree that 0.01 is unlikely to make sense.
I just looked at data from a Davis Vantage Pro2, and it outputs uv index with resolution 0.1. Looking at last summer, at times that appear blue sky, the index changes smoothly at the 0.1 precision, and I don't see the noise that would be present if that precision were goofy. The curve looks reasonable and tracks solar radiation, and tracks theoretical solar radiation as well as measured solar radiation does. Arguably the data precision should be higher, as you don't want to lose information due to representation. So that's at least an existence proof that one device received by rtl_433 is sensibly reporting UV index to 0.1. I can certainly believe that there are some sensors for which this isn't true :-)
The smoothness of the data was never really any concern for me, and I'm sure all other stations with an integer UV Index also have smooth raw data from which they round.
The only thing I was wondering about was that any station presenting a one decimal UV value to the user maybe should also have their UV Index renamed to UVA index or similar, as with any such precision a more precise label might also be suitable, while with any rounded values for the other stations a generic UV is sufficient.
If and when I might get such a station to replace my current one, I'm not too bothered about what current users assume they might be getting with that precision. But when I do get one I'd be very interested in which kind of UV sensor they have built in and what its precise nanometer range and responses are :)
Or more bluntly, like for the FinhrOffset WS90, which seems at least to be honest about its UV accuracy, as most others usually only are with the temperature and humidity.
From the SW90 manual specification section
So what good is a single decimal presentation of a UV value which is only accurate to ±2 - when an integer would also definitely suffice instead, without giving the expectation of a more fine grained more accurate value.
And even your Davis Vantage Pro2, which admittedly is an even more professional unit, states its accuracy only as ±0.5. And it seems to be even more UVB weighted, so my UVA assumption above was wrong, for the Davis at least …
So the integer rounding of most other stations seems sensible to me, as it at least doesn't give the impression of a false accuracy, which isn't really there with even pro-user station UV sensors.
So the integer rounding of most other stations seems sensible to me, as it at least doesn't give the impression of a false accuracy, which isn't really there with even pro-user station UV sensors.
The point of rtl_433 reducing precision is not to convey errors to users. It is to use the smallest precision such that it is pretty clear there is no loss of possibly useful information. Error estimates should be conveyed separately.
And, as I said before, just because there is an accuracy spec, that does not mean that granularity that is finer than the accuracy spec is not useful or inappropriate. It's quite clear that my readings within an hour or two are stable to better than 0.1 UV index units, even if the absolute calibration is unclear.
Most importantly, people use rtl_433 as a way to log data and later analyze it. For that to work, you have to have enough resolution so that you are not losing information in the data transmission pipeline. So unless you are starting to see measurement noise in your data, you don't have enough resolution.
If you want to round to integer for storage or display, by all means do it. But you are arguing that because that's what you want to do, nobody should have access to the higher-resolution data provided by some measurement devices.
A comparable thing is temperature. I have multiple sensors, and some are 0.1 degree and some are 0.01. Yes, it's clearly ridiculous to expect absolute accuracy anywhere near 0.01C. But, when I look at the graphs, I see signal. I can see the heating system cycling with a 0.05F change in temperature (this isn't near a heat source). This isn't an rtl_433 sensor (si7021/esphome), but it would be bad to clamp resolution to even 0.1F for it, as it would harm the data. I say this even if I would not expect two of them to agree to 1F.
No absolutely no capping for anyone and best to pass on the values as being broadcast and received by the stations, but with the very small scale of the UV index, from 0-16, but realistically for most people around the world from 0-10, an accuracy of ±2 within that small scale and at the same time presenting values with a 0.1 precision I find quite quite embarrassing - not on rtl_433's behalf, but from the manufacturers of these stations. And it definitely makes me stick to my 'purely marketing' decision opinion, as most users will take these values at face value, without ever questioning the science behind it.
My opinion is that rtl_433 should be usable out of the box and relay as raw data as possible. Most often what the station receiver displays is what rtl_433 also should output. This does not work always but should be the guiding principle.
If we now look at what Wikipedia writes about UV Index we get this:
The UV index is designed as an open-ended, directly proportional to the intensity of UV radiation, and adjusting for wavelength based on what causes [human skin](https://en.wikipedia.org/wiki/Human_skin) to sunburn.
So by the name index is confusing to me and made me believe that it should have been an integer but is looks like it should be a float and then rtl_433 also should output a float value, even if sensors might output dubious readings.
Patch LGTM.
I've asked the original author of the value to UVI mapping for some input with https://github.com/merbanan/rtl_433/issues/764#issuecomment-2568349370
@DigiH you have the WH24, right? Can you confirm or improve on the derived linear formula to replace the indexed mapping?
I really didn't want to necessarily propose any changes, but by now you should all know I am a stickler when it comes to accuracy and consistency 😜 otherwise I wouldn't have made three tries to get the battery ambiguity and now the UV index on the plate - I know there was at least another inconsistency, but for the life of me I cannot remember what it was, so it'll have to wait until I stumble across it again.
It's just that from my past job and projects I found it best to voice anything which I find peculiar, strange or just a bit unusual. Not so much for any immediate necessary action in all cases, but for everyone to throw in their ideas as well, and especially for the awareness for any future similar implementations and/or occurrences.
you have the WH24, right?
I have the WH65B , or more precisely a Froggit rebrand which gets recognised and decoded as a FineOffset WH65B. So yes, I suppose it is the same implementation as the WH24.
But still, any UV sensor array being used for proper scientific observations costs more than a few of our weather stations put together, and the only time, if at all, a weather station sensor will report a true to actual value is only during the short time of the day when it is, if at all, totally perpendicular to the sun's altitude and azimuth. Much the same as you can calculate the maximum theoretical solar power throughout the day, but your solar panels can only receive and use that maximum during that time of day when they are also directly perpendicular to the sun - unless you constantly tilt and rotate your weather station/solar panels to follow the sun.
I read you comment and yes, the 5029 to 5230 jump looks like a typo, but I'd also not spend too much time in trying to fine tune an already not totally accurate base reading, due to sensor restrictions. And so far the UV Index for my WH65B has been looking fine - not that I monitor it on a daily basis - and luckily there's never been any occurrence of 13 or even 12 in my parts of the world. I'll keep a closer eye on it however.
Can you confirm or improve on the derived linear formula to replace the indexed mapping?
The only other thing I'm getting more and more confused with today is the actual UV Index scale range, which for the WH24/WH65B is 0-13, for the FineOffset WS90 above it's 1-15, the Davis Vantage Pro2 0-16 and the Wikipedia entry stating 0-11+ - open-ended yes on theory, but some starting at 1 instead of 0, and all stations defining a finite index range , at least in their specs.
So it will definitely be interesting to find out how the original 0-13 scale came to be, especially in the light of other FineOffset UV sensors.
Can you confirm or improve on the derived linear formula to replace the indexed mapping?
The only other thing I'm getting more and more confused with today is the actual UV Index scale range, which for the WH24/WH65B is 0-13, for the FineOffset WS90 above it's 1-15, the Davis Vantage Pro2 0-16 and the Wikipedia entry stating 0-11+ - open-ended yes on theory, but some starting at 1 instead of 0, and all stations defining a finite index range , at least in their specs.
So it will definitely be interesting to find out how the original 0-13 scale came to be, especially in the light of other FineOffset UV sensors.
I'm pretty sure there is no upper limit to the index, and to Benjamin's point calling it 'index' when it's really just a scale is confusing. Depending on where you are on the earth, season, etc. the value will vary. Yes, there's probably a "max observed so far" kind of number, and I bet that is what the top end is. Sort like solar radiation is 0 to 1200 W/m^2. In AU/NZ, the values are higher than in the US due to reduce ozone thickness or something like that.
I've definitely see values less than 1 out of Davis, and 0 at night. So I'd say anybody saying it has a lower limit of 1 is just off.
It's remarkably hard to find anything quantitative in a primary source from WHO, US EPA etc. But US National Weather Service says UVI 1 is 25 mW/m^2 of UV, without giving the wavelength range, and mentions "mid teens", which is all consistent with index has no upper bound in theory but has highest observed values. https://www.cpc.ncep.noaa.gov/products/stratosphere/uv_index/uv_compute.shtml
I've also been trying to find more information, and yes, it needs to be open-ended for any space probes, especially solar ones observing and getting closer to the sun.
I also wonder at what index value we couldn't survive at all any longer, knowing that high index unprotected exposure is already very harmful. And with so many sterilisers for phones and gear and hoovers these days implementing high UV radiation to kill germs and bugs, it is quite an interesting subject, but also very elusive.
In that respect I also wonder that, once the 0-13 scale has been confirmed and verified for the WH24 decoder, if the raw uv property would still really be useful to publish. I've always had it present for my WH65B, but always wondered what it actually was, without any identifiable unit, not more really than the encoded index scale.
There are a few different things in this discussion that I want to address (some additional info, some reinforcement of things others have said):
I have a (new/recent) WH65B (Ambient Weather WS2902-ARRAY) and an Ambient Weather "Weather Hub". The UV Index produced by rtl_433 does NOT match the UV Index produced by the "Weather Hub". I'm still working on gathering exact mappings of raw to UV Index values from the "Weather Hub" (and I will update this comment/issue as I learn more), but it looks roughly like raw values 0-99 produce UV Index 0, 100-549 produce UV Index 1, then UV Index increases by 1 for every 450 increase in raw value.
Details on UV Index: https://en.wikipedia.org/wiki/Ultraviolet_index#Technical_definition TLDR: Calculate the Diffey-weighted UV irradiance (or measure irradiance after passing light through an appropriate UV filter) in units of mW/m^2, then divide by 25 and optionally round to the nearest integer to get the UV Index. When the index was designed, it was intended to be practically limited to 10 at the earth's surface, based on peak UV irradiance at the earth's surface at the time. However, there is no upper limit, and current peak UV irradiance at the earth's surface is higher than the 10 level.
Note that the above two paragraphs imply that the raw UV value transmitted by the WH65B is the Diffey-weighted UV irradiance in increments of 0.055mW/m^2, or is the UV Index in increments of 0.0022 I'm fairly certain that the published "0-13" range for the WH24/WH65B simply means that the sensor won't read (or won't accurately read) UV irradiance above the level of a 13 UV Index.
Regarding the resolution of values provided by rtl_433: I can confirm (using a UV lamp) that the raw UV values produced by the WH65B are very precise (reproducible and sensitive); the values appear consistently reproducible given a fixed input irradiance, and the sensor can reliably detect very small changes in the input irradiance throughout the range I can test (~0-3333 raw values). This does not necessarily mean the values are accurate; I can't speak for how well these values align with the actual UV Index standard or a calibrated instrument. Rounding the UV Index (calculated from the WH65B's raw UV value) to 0 or 1 decimal places loses a LOT of precision. There may be use cases where precision is useful/important and accuracy to the standard is not important. For example, the UV and visible light sensors might be used to detect cloud density based on relative and time-based measurements at a particular location, in which case a precise measurement is important, although accuracy to an irradiance standard is probably irrelevant. Therefore, my opinion is that rtl_433 should either provide a full-precision UV Index value (which users can subsequently round in other software if desired), or rtl_433 should continue to provide separate full precision (either raw, or converted to DUV or UV Index with full precision) and rounded (UV Index) values to support different use cases. In other words, I have no strong opinion on whether one or two values is provided, but if only one value is provided, then my opinion is that it should not be rounded, even if the extra precision is useless for most use cases.
Generally, rtl_433 doctrine is to round to the actual resolution of the device. UV Index appears to be a real number kind of quantity, and for a device that receives a weighted irradiance with highish resolution, it makes sense to me to convert to UVI and then round such that less rounding would be silly. That sounds %.3f as the right blend of enough precision and not goofy. Is that what you are thinking?
Of course, some other device might well have different characteristics. For example, Davis Vantage Pro2 units seem to output UV with 1 decimal digit, %1.f. That's not really an rtl_433 thing, and it shouldn't affect other devices. Just saying that not all UVI will be high resolution.
If we agree on the "uvi" key to be double then we don't really need to mind the precision. The human-readable kv output can specify a fitting format in each decoder and the machine-readable formats (jsonl) have 3 decimals anyway.
My confusion is the same as @DigiH in the first comments, I imagined UV Index as a classification system with discreet steps. But if some sensors present decimals then we don't want to truncate that.
The main thing here is to normailze UV Index keys to e.g. "uvi", "UV Index", DATA_DOUBLE, … -- I guess this change can be applied?
@PaulSD excellent analysis then should go in the WH65B documentation and I also like the change to the linear mapping with the 0.0022 factor.
I think it's been argued well that it's just a real number. I'm +1 for the normalization of names you describe.