Rainfall reading spikes with Acurite-Rain899 rain gauge

[deviantintegral]

I recently set up an Acurite 0899 rain gauge, and I am getting spurious readings every few days from it:

[weather_station] time : 2022-07-07T01:19:57-0400 [weather_station] model : Acurite-Rain899 [weather_station] id : 3248 [weather_station] channel : 0 [weather_station] Battery : 1 [weather_station] Rainfall Accumulation: 4.83 mm [weather_station] Integrity : CHECKSUM [weather_station] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [weather_station] [weather_station] time : 2022-07-07T01:19:57-0400 [weather_station] model : Acurite-Rain899 [weather_station] id : 3248 [weather_station] channel : 0 [weather_station] Battery : 1 [weather_station] Rainfall Accumulation: 645.92 mm [weather_station] Integrity : CHECKSUM

[weather_station] time : 2022-07-07T02:45:58-0400 [weather_station] model : Acurite-Rain899 [weather_station] id : 3248 [weather_station] channel : 0 [weather_station] Battery : 1 [weather_station] Rainfall Accumulation: 645.92 mm [weather_station] Integrity : CHECKSUM [weather_station] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [weather_station] [weather_station] time : 2022-07-07T02:46:57-0400 [weather_station] model : Acurite-Rain899 [weather_station] id : 3248 [weather_station] channel : 0 [weather_station] Battery : 1 [weather_station] Rainfall Accumulation: 4.83 mm [weather_station] Integrity : CHECKSUM

What's interesting is that at least in the last few cases, the amount read is 645.92 mm. The display unit does not show these values at all, so either rtl_433 is misinterpreting the signal or it has some sort of built-in filter.

Is this a common issue? I'm wondering if this should be filtered out in rtl_433 or if the reading is actually "correct" and I should filter it out downstream.

[zuckschwerdt]

Rainfall here is just two bytes (7 bits each only): https://github.com/merbanan/rtl_433/blob/master/src/devices/acurite.c#L998-L1002 Likely a bad reading the checksum didn't catch. Use -R 40:v to show the raw message bits.

[gdt]

There's a larger issue beyond the scope of this ticket, which is that rtl_433 by design just decodes frames from the radio and outputs them. In an ideal world all devices would have robust CRCs or ECC that results in ~zero bad decodes. But also ideally if there are repeated transmissions that are supposed to be the same, there would be some soft decoding across all of them to get a single best estimate. All of this is very hard, and of course not about this decoder. Right now I think the best approach is to have some processing code that consumes the json from rtl_433 and does some sort of rejection of outliers and also somehow processes the multiple transmissions as a group.

I recently learned that home assistant has facilities to filter surprising values; this is the sort of thing I am talking about but I think it would be nice to have it as part of rtl_433 the project (not necessarily the program), either json->json, or as part of json->mqtt. Perhaps a python library includable in both.

[rct]

I recently learned that home assistant has facilities to filter surprising values;

@gdt - pointer please? I've been looking for something like that. And since my topology is

rtl_433 -> MQTT -> Home Assistant -> InfluxDB, etc. 

it would help to filter values before Home Assistant.

All of this is very hard, and of course not about this decoder.

Agreed.

Right now I think the best approach is to have some processing code that consumes the json from rtl_433 and does some sort of rejection of outliers and also somehow processes the multiple transmissions as a group.

Also agreed. Different consumers/different applications (weather/environment sensors vs. security, flood, other loss prevention) have different requirements.

this is the sort of thing I am talking about but I think it would be nice to have it as part of rtl_433 the project (not necessarily the program), either json->json, or as part of json->mqtt. Perhaps a python library includable in both.

I've been thinking about:

rtl_433 -> MQTT -> ... context aware filtering application ... -> MQTT -> all consumers.

Loggers and other things interested in the raw stream could get it off the raw MQTT messages. Things that need/want cooked/processed/filtered could read the reprocessed messages.

[gdt]

I think it was https://www.home-assistant.io/integrations/filter/ This would help me, were I to configure it, because i have an Acurite 986 fridge/freezer temp sensor, and the freezer one works well but I see bad values (-30C) on the fridge one.

I agree that requirements are different per domain. But what I meant is that if a given transmitter sends 4 copies of the message, presumably on the theory that pulse or occasional transmitter interference won't clobber them all, then a receive processor that looks at all 4 and figures out "given what I heard, what is the message most likely to have been sent" is useful, regardless of whether it is "door open" or "fridge is 2C". But I can see that even a low odds of "door open" should perhaps be reported.

rtl_433 -> MQTT -> ... context aware filtering application ... -> MQTT -> all consumers.

I can see why you'd want that, but I view the filtering as part of what a receiver for this sort of signal ought to be doing. I know rtl_433 has a frame in, line out mentality, and I'm not trying to argue for deviating from that, but if you think about a receiver for these signals from a system viewpoint, it really should be treating multiple frames in a burst, and past data, in a holistic manner. I would prefer to do that pre-mqtt, which I view -- and i know this is highly arguable -- as a bus for valid data.

[peterchs]

Node red is also an option for resolving this issue.

e.g. rtl_433 -> MQTT -> node red (filter or aggregate or interpret data) -> influxdb or other consumers

I'm fine with this and wouldnt expect complex state or debouncing or any such processing to be done within rtl_433, primarily as such processing can vary so much dependent on device and usage of the data.

[deviantintegral]

Thanks for all the feedback. I've enabled verbose mode for protocol 40, and I expect I'll have a frame in the next few days.

For myself, I'm perfectly happy to add a filter via Home Assistant. However, I had two thoughts about improving the UX assuming we determine that rtl_433 is decoding a "valid" frame:

1. When I see the line Integrity : CHECKSUM, it gives me confidence a valid frame was decoded. If this, and other protocols, have weak checksums where collisions can occur every few days, then I think the rtl_433 output should be clear about that.
2. If we determine that the transmitter is sending these frames with the large readings, and there's logic in the display unit filtering them out, then I would agree that rtl_433 is the wrong place to do the filtering. After all, there may be many different ways to filter the data to get a good value, especially when you consider something like a battery level. But, could this be a good opportunity to document and show example filters in the examples directory?

[gdt]

For myself, I'm perfectly happy to add a filter via Home Assistant. However, I had two thoughts about improving the UX assuming we determine that rtl_433 is decoding a "valid" frame:

I don't follow what valid means, with or without quotes. The transmitter has bits to transmit, and they are encoded in RF, which goes over a noisy channel, and then the receiver gets waveform measurements, and a software receiver converts that to bits, and then there is checksum/etc. processing. rtl_433 outputs bits when the bits that were decoded pass checksum processing. That doesn't mean they are the same bits that the transmitter tried to send.

1. When I see the line `Integrity : CHECKSUM`, it gives me confidence a valid frame was decoded. If this, and other protocols, have weak checksums where collisions can occur every few days, then I think the rtl_433 output should be clear about that.

I think that is the nature of checksum. Given the embedded nature of these devices, when I see checksum I think 16 bits or 8 bits. If you have a frame and apply arbitrary corruption (so you get random bytes instead of sent bytes), then for 8 bit checksum the odds of passing are 1 in 256. For 16 bits it is 1 in 65536. A checksum is just not that great at keeping out bad data, and not trying to be mean, but your confidence is misplaced :-)

2. If we determine that the transmitter is sending these frames with the large readings, and there's logic in the display unit filtering them out, then I would agree that `rtl_433` is the wrong place to do the filtering. After all, there may be many different ways to filter the data to get a good value, especially when you consider something like a battery level. But, could this be a good opportunity to document and show example filters in the `examples` directory?

Likely the transmitter is not sending them. You could set up two receivers, one a few feet from the transmitter and once someplace where only 50% of the frames decode, and then compare them..

Where the right place to filter has no crisp answer; it's a system design question. Certainly if you write a filter that's useful to you putting it in examples seems good. I think filters to be optimal will need to be device-aware, knowing both how many copies of frames are sent, and about the semantics of the transmitted data.

[zuckschwerdt]

Quick note on the "mic" field: We have

• "PARITY" : a few bits of integrity check, 1 or 2 bits strength
• "CHECKSUM" : weak byte of integrity, maybe about 4-6 bits true strength
• "CRC" : strong byte of integrity, 8 bits or more true strength

Arguably we could output that number, or more steps like NIBBLE and CRC16, but it's hard for authors to pick that correctly. This none/weak/good/strong scheme seems detailed enough.

https://triq.org/rtl_433/DATA_FORMAT.html#message-data

[gdt]

The size is important; 8-bit CRC is still not great. Perhaps CHECKSUM and CRC should be defined to be 8 bits, and we would use CRC16 if that's the case. But I'm guessing ~all protocols are just 8 bits.

I think we really do need to ponder how best to use the redundant information in repeated transmissions. However I have seen Ford TPMS that seems to have one of the 4 have a different pressure value by 0.25 psi, which feels like measurement noise, not an error.

[deviantintegral]

I don't follow what valid means, with or without quotes.

I meant that the transmitter could be sending bad data, even with a checksum passing, that logically was incorrect. In that case, I don't think rtl_433 can or should know how to handle the frame and shouldn't filter or change it.

Quick note on the "mic" field:

That's great context. I think that's worth adding to the docs, so I'll file a PR for that.

A checksum is just not that great at keeping out bad data, and not trying to be mean, but your confidence is misplaced :-)

Fair enough! A bad entry happened again. Unfortunately I think I have an unrelated bug in my output buffering, so I can't 100% correlate stdout and stderr log message ordering:

stdout:

time      : 2022-07-19T07:09:51-0400
model     : Acurite-Rain899
id        : 3248
channel   : 0
Battery   : 1
Rainfall Accumulation: 5.08 mm
Integrity : CHECKSUM
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

time      : 2022-07-19T07:09:51-0400
model     : Acurite-Rain899
id        : 3248
channel   : 0
Battery   : 1
Rainfall Accumulation: 678.69 mm
Integrity : CHECKSUM

stderr:

acurite_txr_decode: Acurite 5n1 raw msg: {64} 3c b0 f0 00 00 00 14 f0
acurite_txr_decode: Acurite 5n1 raw msg: {56} 3c b0 f0 00 00 14 f0
acurite_txr_decode: bad checksum: {63} 3c b0 f0 00 00 00 29 e0
pulse_slicer_pwm(): Acurite 592TXR Temp/Humidity, 5n1 Weather Station, 60 Atlas
bitbuffer:: Number of rows: 3
[00] {64} 3c b0 f0 00 00 00 14 f0
[01] {56} 3c b0 f0 00 00 14 f0
[02] {63} 3c b0 f0 00 00 00 29 e0

So if I read this right:

• Row 0 is a correct frame, corresponding to 5.08mm
• Row 1 fails the checksum and is not output.
• Row 2 is the checksum-passing but invalid frame reporting 678.69mm

I thought I could verify this with rtl_433 -R40 -y '{63} 3c b0 f0 00 00 00 29 e0' but that's not printing any output.

Related: https://github.com/merbanan/rtl_433/issues/1964

[zuckschwerdt]

That's inverted data, you'd want rtl_433 -R40:vv -y 'c34f0fffffffeb0f/ c34f0fffffeb0f/ c34f0fffffffd61f'

[anthyz]

I took a look. It's the row 2 message that isn't output. The row 1 message appears to drop a zero byte from the middle. Because it is zero the checksum calculation is the same and passes. However the decoder does not verify length for 5-n-1 messages, and incorrectly it decodes the raincounter from the wrong values because the last two bytes have moved up.

https://github.com/merbanan/rtl_433/blob/10891fb732e65f3c6962a72ac05dd9656f07638e/src/devices/acurite.c#L1002

[gdt]

Definitely change the code to verify length, if the length is supposed to be fixed. Particularly with less than a 16-bit CRC (ha!) false decodes are possible and IMHO everything you can do to reject bad decodes is progress.

[deviantintegral]

Thanks for the investigation. I started a PR at https://github.com/merbanan/rtl_433/pull/2116.

[deviantintegral]

I made good progress on testing this today, and addressed the review notes. In the mean time, I thought given the original issue (600mm in one reading!) that https://what-if.xkcd.com/12/ would be of interest ☔ .

[rct]

Are there are sample files available for the 899?

I didn't see any in rtl_433_tests. I'm trying to do some testing for some potential cleanup.

[deviantintegral]

@rct Looks like I typo'ed the name at https://github.com/merbanan/rtl_433_tests/pull/428 and it should be 899. I'll update it now.

[deviantintegral]

This is fixed at https://github.com/merbanan/rtl_433/pull/2116, but the tests PR at https://github.com/merbanan/rtl_433_tests/pull/428 is still open. Anything more for me to do there?

[zuckschwerdt]

tests PR still open. Anything more for me to do there?

No, all good. We are very much behind on sorting and merging tests data though. I need to work through all those someday…

[gdt]

Calling this partly a discussion and partly now listed as a use case in #2635