Processing Wireless M-Bus signals

[PovilasID]

trafficstars

Current Situation

Hey,

I have TTGO Lora 32 V2.1.6 flashed with fsk version of OpenMQTTGateway and Nooelec RTL-SDR both listening at 868.95Mhz but one parsing signal from water meter using Wireless M-Bus. I was getting weather station signals on 868.3Mhz just fine, so I was relatively confident that it should work but made sure I used the handy guide to compile a latest firmware option just with a tag to publish unparsed. Just to make sure that HW is capable of capturing the signal.

So in the logs just copied out from RTL-SDR dongle running 'vanila' rtl_433 and successfully capturing and decoding the signal in Debian docker container and just bellow what I suspect a log message for capturing but failing to process on esp32 running rtl_433_esp.

What do you suggest trying?

Logs

3245501
time = 2024-06-10 14:15:05
protocol = 104
mode = T
M = BMT
id = 3245501
version = 23
type_string = Warm Water
C = 68
data_length = 37
data = 2244b4090155240317068c00487ae80000000c1340450700046d072d0a360f0300000000000ba3
mic = CRC
CI = 140
AC = 0
ST = 0
CW = 0
min_volume_flow_min_0 = 1426131.380 m3/min
min_energy_wh_0 = 9176599.000 Wh
mod = FSK
freq1 = 868.92947
freq2 = 868.9719
rssi = -0.516766
snr = 22.8193
noise = -23.3361

{"model":"undecoded signal","protocol":"signal parsing failed","duration":3996,"rssi":-100,"pulses":18}

Configuration

[env:lilygo-rtl_433-fsk]
platform = ${com.esp32_platform}
board = ttgo-lora32-v21
; ~/.platformio/packages/framework-arduinoespressif32/variants/.../pins_arduino.h
board_build.partitions = min_spiffs.csv
lib_deps =
  ${com-esp32.lib_deps}
  ${libraries.ssd1306}
  ${libraries.rtl_433_ESP}
build_flags =
  ${com-esp32.build_flags}
; *** OpenMQTTGateway Config ***
  ;'-UZmqttDiscovery'          ; disables MQTT Discovery
  '-DvalueAsATopic=true'    ; MQTT topic includes model and device
  '-DGateway_Name="OMG_lilygo_rtl_433_ESP_FSK"'
  '-DOOK_MODULATION=false' ; FSK modulation activated
  '-DRF_FREQUENCY=915'
  ;-DRF_FREQUENCY=868.300'
  ;-DRF_FREQUENCY=433.9'
; *** rtl_433_ESP Options ***
;  '-DRTL_DEBUG=4'             ; rtl_433 verbose mode
;  '-DRTL_VERBOSE=58'          ; LaCrosse TX141-Bv2, TX141TH-Bv2, TX141-Bv3, TX141W, TX145wsdth sensor
;  '-DRAW_SIGNAL_DEBUG=true'   ; display raw received messages
;  '-DMEMORY_DEBUG=true'       ; display memory usage information
  '-DDEMOD_DEBUG=true'         ; display signal debug info
;  '-DMY_DEVICES=true'         ; subset of devices
  '-DPUBLISH_UNPARSED=true'    ; publish unparsed signal details
;  '-DRSSI_THRESHOLD=12'       ; Apply a delta of 12 to average RSSI level
;  '-DAVERAGE_RSSI=5000'       ; Display RSSI floor ( Average of 5000 samples )
  ; '-DSIGNAL_RSSI=true'         ; Display during signal receive
  ; '-DNO_DEAF_WORKAROUND=true'
; *** OpenMQTTGateway Modules ***
  '-DZgatewayRTL_433="rtl_433"'
  '-DZradioSX127x="SX127x"'
; *** ssd1306 Display Options ***
  '-DZdisplaySSD1306="LilyGo_SSD1306"'
;  '-DLOG_TO_OLED=true'         ; Enable log to OLED
;  '-DJSON_TO_OLED=true'
;  '-DLOG_LEVEL_OLED=LOG_LEVEL_NOTICE'
;  '-DDISPLAY_IDLE_LOGO=false'
;  '-DDISPLAY_BRIGHTNESS=80'
;  '-DDISPLAY_METRIC=false'
custom_description = For ESP32, Gateway using RTL_433_ESP and RadioLib with FSK modulation (beta)
custom_hardware = ESP32 LILYGO LoRa32 V2.1

Environment

Process Supervisor

not applicable

Additional Context

No response

[NorthernMan54]

With FSK reception we have found it may need a more precise frequency than the rtl_sdr. Try adjusting the frequency slightly

[PovilasID]

With FSK reception we have found it may need a more precise frequency than the rtl_sdr. Try adjusting the frequency slightly

I tested it by listening on every rf frequency from 868.91 to 868.99 Mhz adding 10 khz and giving it some time. However, the most unparsed signals I received were at 868.95 Mhz as per spec, so I really do not think that the issue is in capturing the signal. Any more suggestions?

[Entropy512]

Do you know which mode your meter is using?

For modes S, T, and C&T, the bitrates are quite high and as a result you might be missing the preamble - see my comments at the end of https://github.com/NorthernMan54/rtl_433_ESP/issues/83

Check out my proof of concept at https://github.com/Entropy512/rtl_433_ESP/commits/neptune_r900/ - but you'll need to change the bitrate setting for Mode C&T, and also you'll need to change the MY_DEVICES defines to include wireless mbus.

[PovilasID]

@Entropy512 It is running in T mode. WMBus also referred to as T1. How did you figure out the correct MINIMUM_PULSE_LENGTH and MINIMUM_SIGNAL_LENGTH ? Trial an error? Is there a verbosity option on vanilla rtl_sdr that provides those details?

Also in the repo you defined DOOK_MODULATION=true shouldn't it be false for FSK signals?

[Entropy512]

@Entropy512 It is running in T mode. WMBus also referred to as T1. How did you figure out the correct MINIMUM_PULSE_LENGTH and MINIMUM_SIGNAL_LENGTH ? Trial an error? Is there a verbosity option on vanilla rtl_sdr that provides those details?

Read the source for the decoder. For example I was using neptune_r900: https://github.com/NorthernMan54/rtl_433_ESP/blob/main/src/rtl_433/devices/neptune_r900.c#L238

Choose MINIMUM_PULSE_LENGTH that is lower than the shortest of those two (if different). I chose 25 in case of some jitter, although with the bit synchronizer it's probably fine. Without the bit synchronizer you likely need to drop that even more. I've seen pulses as low as 20 without it.

That struct doesn't directly give you the minimum signal length - you can calculate that by multiplying the pulse length by the number of bits, which is USUALLY in the comments (168 bits for Neptune, although that's after preamble and syncword)

Right now MINIMUM_SIGNAL_LENGTH is also reused as the minimum gap length (which IS listed in that struct), this probably needs to be fixed. I'm positive I made a comment about this before but I can't find it... I wonder if I typed something, didn't click update/post, and then closed the browser window...

Also in the repo you defined DOOK_MODULATION=true shouldn't it be false for FSK signals? Yeah you'll need to change that too for wireless Mbus - here on the wrong side of the pond, Itron and Neptune both used OOK. (Itron has moved to FSK with their latest gen, but it's also encrypted now, so unless your utility sets it up to connect to your home wifi and gives you developer certs, you can't read it. Xcel Energy did this, NYSEG has not.)

[PovilasID]

@Entropy512 For M-bus T mode it looks like preamble has a hardcoded preamble https://github.com/NorthernMan54/rtl_433_ESP/blob/b8ea0e2d8bada6c6cde31640481c80fe674294a2/src/rtl_433/devices/m_bus.c#L953 So it should not be a problem. Minimum signal length is still potentially a problem. In FSK mode it is specified as 500 https://github.com/NorthernMan54/rtl_433_ESP/blob/b8ea0e2d8bada6c6cde31640481c80fe674294a2/src/rtl_433_ESP.h#L96 How ever in the device decoder it seams to be dynamically calculated: https://github.com/NorthernMan54/rtl_433_ESP/blob/b8ea0e2d8bada6c6cde31640481c80fe674294a2/src/rtl_433/devices/m_bus.c#L963 Am I right or am I seeing two different things that vaguely similar and equating them falsely? I have no clue when it comes to signal processing.

[Entropy512]

As to the preamble - that is the problem. The preamble occurs at the beginning of the message, and without my change, if that comes in before the RSSI rises, it's ignored by the interrupt handler. Then the message without a preamble is passed to the decoded and the decoder ignores it because it doesn't see the preamble/syncword. It looks like all of my message lengths are a multiple of 1 ms so I'm guessing that's the RSSI polling rate, so if your message is 4.15 milliseconds long, you could lose as much as the first 1/4 of the message.

At least for messages with preambles/sync words, my change shouldn't cause any negative issues - any garbage that precedes the message will be ignored unless it miraculously looks exactly like a preamble + sync pattern, which is extremely unlikely.

There might be some simplistic protocols with slow bitrates and no preamble though? These might have negative consequences from my change. On the other hand you should have multiple RSSI polling intervals per bit in those cases.

32+138 = 136. 136(1/32768) = 0.00415 seconds per message. That's shorter than the minimum message length for R900 (168 bits plus header/preamble)

If you're trying to receive that 100 kbit/sec mode, that'll be 1/3 the length.

[PovilasID]

@Entropy512 I appreciate your help do not really have the time to learn signal processing, so if you could be more specific in what do you think could work it would be grate.

[Entropy512]

At least since most of the WMBus modes are the same bitrate as R900: Start with my proof-of-concept patchset Disable OOK Either disable MY_DEVICES, or add Wireless MBus to the decoders supported by MY_DEVICES (look at my patch) Drop MINIMUM_SIGNAL_LENGTH a little. Looks like 4.15 milliseconds (see my previous post) = 4150 microseconds, I'd take 80% of that. I think it's already shorter than that for FSK????

You'll also have to change the bitrate for FSK since that's a separate code block than the one I changed for OOK. You have the same value (32.768 kbps), it just needs to be changed for FSK devices.

[NorthernMan54]

I refreshed the device decoders from rtl_433 24.10, maybe this is resolved?

[PovilasID]

@NorthernMan54 m bus files do seam to be modified on the latest update but in rtl433 change log there was a mention fix of length calculations, so there is a chance. I will try to test it on Monday.

[PovilasID]

@NorthernMan54 Update after testing latest version: TL;DR Did not seam to fix the issue. I used a web flasher to update my firmware setting it to lilygo-rtl_433-fsk. Then I ran noolec rtl-sdr dongle with vanilla rtl_433 running parallel. I set both to same frequency of 868.900. rtl-sdr dongle is capturing wmBus packets but lilygo running OpenMQTT rtl_433 is not. I will run for 24 hours but I do not expect to see a changes

[NorthernMan54]

OMG has not been updated to 0.4.0 yet, still in testing

See this - https://github.com/NorthernMan54/rtl_433_ESP/issues/150#issuecomment-2516895680

[PovilasID]

@NorthernMan54 Sorry for not checking if OMG has been updated. I used your dev build of OMG to flash the 81eb8b commit build. However, still no captures. Will leave it running for longer time and if your are interested in any logs let me know.

[PovilasID]

I saw that rtl_433 has a new release that changes some m-bus processing. @NorthernMan54 Are there plans to update this repo with new 25.02 version? Timeline? No pressure just curious. However I do entirely expect an update to fix this issue because maintainer of this repo has mentioned that he mostly updates only the decoders not signal processing https://github.com/NorthernMan54/rtl_433_ESP/issues/160#issuecomment-2500589289 and from what I understood from other comments that is where the problem is at. I do not know enough to fix it but if somebody has a fork with a possible fix I would be happy to do some testing and validation.

[jaal2001]

Are there any updates on this? I do have the same setup. @PovilasID Do you have any repo with the appropriate compiler settings? So far I did not manage to compile a running version.

[PovilasID]

@jaal2001 It has been a while but as far as I remember I would for this repo make some changes and then replace rtl_433_ESP with my own repo in OpenMQTTgateways dependency list and from there I would follow the web compilation instructions. https://docs.openmqttgateway.com/upload/gitpod.html After doing some debugging I determined that issue is somewhere in signal processing that is not update as versions get updated, so because of this decoupling I am not sure if issue is getting fixed if there is no significant re-write or maybe I am just mistaken. Either way it is a little above my level of understanding to deal with it myself.