HmIPW-DRAP/HmIP-HAP cannot identify RaspberryMatic as a HA addon

[ThomasSiegmund]

Running RaspberryMatic in a standard Docker subnetwork on the host prevents a DRAP from connecting ...

Steps to reproduce the behavior

1. Configure a DRAP to connect to a standalone RaspberryMatic. Make sure it can connect and the LED lights up in turquoise
2. Switch off the DRAP
3. Move your RaspberryMatic installation into a Docker container, e.g. by creating a backup and restoring from it in a freshly set up Docker installation according to the installation guide. I used the docker-compose variant of the docker setup.
4. Power up the DRAP The DRAP goes to the usual sequence of yellow, green and turqoise lights. In the end it flashes in turquoise, indicating that it wants to connect to CCU but does not find it.

Expected behavior The DRAP should discover the CCU and connect to it.

System information:

• RaspberryMatic Version: 3.59.6.20210807
• Host OS: Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1041-raspi aarch64)
• Used Hardware: Raspberry 4
• Used HomeMatic RF-Module: RPI-RF-MOD on HB-RF-USB-2

Additional context The issue seems to be related to network discovery by UDP broadcasts. I noticed that RaspberryMatic running on bare metal is visible to EQ3's Netfinder program, but RaspberryMatic running under Docker ist not. So I watched network traffic during Netfinder discovery.

In this example Netfinder is on 10.10.15.1, and two DRAPs are running on 10.10.15.100 und 10.10.15.101

No.	Time	Source	Destination	Protocol	Length	Info
4	0.648462575	10.10.15.1	255.255.255.255	UDP	52	43537 → 43439 Len=10
6	1.348330504	10.10.15.100	255.255.255.255	UDP	103	43439 → 43537 Len=61
7	1.410798515	10.10.15.101	255.255.255.255	UDP	103	43439 → 43537 Len=61

You can see Netfinder sending a UDP broadcast package, which is immediately answerded by the two DRAPs. I guessed that a similar mechanism might also be used by the DRAP at startup to find the CCU. However, this will fail on a standard Docker setup, because Docker does not support this kind of broadcast messages. The only easy way to fix this I know of, is to run the container in host mode like with this docker compose file:

version: "3.7"
services:
  raspberrymatic:
    image: ghcr.io/jens-maus/raspberrymatic:latest
    container_name: ccu
    hostname: homematic-raspi
    privileged: true
    restart: unless-stopped
    stop_grace_period: 30s
    network_mode: host
    volumes:
      - ccu_data:/usr/local:rw
      - /lib/modules:/lib/modules:ro

Here network_mode: host makes the container use the host network without any internal network or port mapping. Just adding this line to the docker-compose.yaml and restarting the container makes the DRAP connect and to go from flashing to solid turquoise light.

Interestingly, in my experience this problem exists only for the DRAPs. HAPs when run as a range extender to a CCU seem to have a different mechanism of network discovery.

[jens-maus]

Thanks for this nice elaboration, this is highly appreciated.

After some investigation, this (as you outlined) can be quite easily solved with adding network_mode: host in the docker/OCI use case. And I think to "solve" it for native docker/OCI users, we simply have to tune the documentation to some extend and could even adapt the deploy.sh script to allow to run docker using the necessary --net=host option so that the container will use the host network accordingly.

However, the same limitation exists in the HomeAssistant addon use case in principle, but seems to be quite harder to solve, I am afraid. While in principle, the HA addon could also be switched to the same network_mode: host mode using a home assistant addon config setting (see https://developers.home-assistant.io/docs/add-ons/configuration/#add-on-config), this could immediately negatively affect other addons running on the same HomeAssistant host. In addition, the internal RaspberryMatic iptables firewall could then quite easily mixup/break the whole host firewall setup if the RaspberryMatic HA addon will run on the host network of HomeAssistant. AFAIK, there is unfortunately indeed no solution to get a docker container to accept UDP broadcast messages that are received by the actual docker host. Perhaps @pvizeli or some other HA dev has some other idea how to forward UDP broadcast packages to the docker containers in HA without having to set host_network: true in the config.json addon config file. Otherwise, I am currently quite puzzled how to consistently solve this for the HA addon use case.

[pvizeli]

If you know the port, then you just need to set it. If it's multicast, then it could pick-up in future by the new multicast plugin

[jens-maus]

Thanks @pvizeli for the hints. Where exactly do I have to set the multicast port? In the config.json of the addon? Do you have some example you can show? And when will the new multicast plugin be part of HA or does a HA user have to install it himself to get udp multicast routing into a HA addon going?

[TVR-X]

Is it possible to test a modified config.json file with "network_mode: host" inside? I can't find a way to modify this on a Addon from a repository because i think this is the same issue like in #1327.

[jens-maus]

Is it possible to test a modified config.json file with "network_mode: host" inside? I can't find a way to modify this on a Addon from a repository

Sorry, but it is not possible for a user to override the network_mode: host for a published add-on. I, as a add-on developer would have to do that in the config.json of the add-on itself. However, I actually tested this myself and just using network_mode: host will end up in other issues that you seeing because of the add-on generally running on the host network (e.g. the network firewall will be modified by the add-on). Thus, I am still waiting for @pvizeli to answer my question and help in getting this multicast issue somewhat solved in the near future.

[pvizeli]

You can asign the port on options but set it to null for the host, so people can just enable it if they enter the port manual. If it using multicast, then it should be solved with https://github.com/home-assistant/plugin-multicast/issues/17

[jens-maus]

You can asign the port on options but set it to null for the host, so people can just enable it if they enter the port manual.

This is already the case, indeed. However, this alone does not seem to help in the end as this issue here shows.

If it using multicast, then it should be solved with home-assistant/plugin-multicast#17

Sorry to bother you, but where exactly does pimd to be installed so that the multicast routing from the host to/from the add-on container works? I am really lacking some more detailed information/know-how here to get this working with HA. As said, right now the relevant udp multicast port (43438) is already defined null in the config.json file: https://github.com/jens-maus/RaspberryMatic/blob/master/home-assistant-addon/config.json#L78

But it seems this alone does not help in getting the multicast traffic from the HA host to the add-on container and vice versa. So any more detailed hint in getting the udp multicast traffic properly routed between the HA host and the add-on container would be highly appreciated.

[dkadioglu]

I am not sure, if the following is of help or does have any short comings, so I just would like to describe my approach and I would be happy, if it helps as well as about feedback. I have a HMIP-HAP working with Raspberrymatic in a Docker container on a RPi4 without using network_mode: host. Instead, I am using the macvlan driver with a exclusive network for the Raspberrymatic container.

This is my docker-compose file:

version: "3.8"
services:
  raspberrymatic:
    image: ghcr.io/jens-maus/raspberrymatic:latest
    container_name: dkhm
    hostname: dkhm
    privileged: true
    restart: unless-stopped
    stop_grace_period: 30s
    volumes:
      - /home/alarm/dkhm_data:/usr/local:rw
      - /lib/modules:/lib/modules:ro
      - /run/udev/control:/run/udev/control
      - /mnt/storage/HMBackup:/mnt:rw
    networks:
      default:
        ipv4_address: 192.168.1.221
networks:
  default:
    external:
      name: hmnet

Before this works, I created the following network. It is important, that subnet is the same as the hosts network and to take care, that ip-ranges of the host network and the to be created Docker network do not collide:

docker network create -d macvlan -o parent=eth0 \
  --subnet 192.168.1.0/24 \
  --gateway 192.168.1.1 \
  --ip-range 192.168.1.220/30 \
  --aux-address 'host=192.168.1.222' \
  hmnet

To allow communication between the host and the Raspberrymatic container, the following additional steps are necessary - as the Docker firewall configuration seems to block this, as far as I understand it:

ip link add hmnet-shim link eth0 type macvlan mode bridge #create a virtual network device hmnet-shim bound to eth0

ip addr add 192.168.1.222/32 dev hmnet-shim #set the above mentioned (aux-address) ip address of hmnet-shim
ip link set hmnet-shim up

ip route add 192.168.1.220/30 dev hmnet-shim protocol static #add the necessary static route between host and hmnet-shim

As the above four lines don't survive a reboot, I created a systemd service for this:

[Unit]
Description=Create hmnet device and add static route
Requires=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c '/usr/bin/ip link add hmnet-shim link eth0 type macvlan mode bridge && /usr/bin/ip addr add 192.168.1.222/32 dev hmnet-shim && /usr/bin/ip link set hmnet-shim up && /usr/bin/ip route add 192.168.1.220/30 dev hmnet-shim protocol static'

[Install]
WantedBy=multi-user.target

With this approacv, the communication between Raspberrymatic and the HMIP-HAP works as it should. The turquoise led is solid and not blinking. The only thing I didn't test is, if the setup process (in german "Anlernen") between Raspberrymatic and HMIP-HAP works, as I did that with network_mode: host and I can't redo the process currently. Maybe somebody else can give feedback, if the setup process works with macvlan, too.

I hope this helps.If you've got questions, please ask and I try to follow up timely.

[jens-maus]

FYI: I just tested the issue myself again and tried to use eQ-3 NetFinder to identify a running RaspberryMatic CCU as a HomeAssistant add-on. As soon as I specify port 43438/udp in the add-on config to be forwarded to the add-on NetFinder is perfectly able to identify the RaspberryMatic also within a HomeAssistant environment. See:

The only thing that is missing/strange is, that this virtual CCU is listed with "refer to help" in the reachability column. This might be, however, due to the fact that this virtual CCU cannot be pinged/reached directly form the system running thie eQ-3 NetFinder tool.

Regarding reachability of the virtual CCU from a HmIPW-DRAP or HmIP-HAP, the situation can be easily similar and needs to be tested. Note, however, that for the DRAP to identify/find your CCU you would have to open port 43439/udp as well.

[THorst92]

Hi @jens-maus,

I don't know if you've received already a feedback to the reachability of the virtual CCU from a HmIPW-DRAP or HmIP-HAP. I've tried to connect a HmIP-HAP to RaspberryMatic which runs as AddOn in Home Assistant without success.

SW-Version:

• RaspberryMatic 3.61.7.20220226
• HMIP-RFUSB 4.4.16
• HMIP-HAP no new update available (checked with the HomeMatic IP App)

In RaspberryMatic all ports are open. The ports 9293, 43438 and 43439 are activated in the configuration of the RaspberryMatic AddOn in Home Assistant.

When I reset the HMIP-HAP twice and try to activate the pairing mode, the LED stays static blue (or yellow if I block the internet connection). In RaspberryMatic I was not able to teach in the HMIP-HAP (local and with internet connection). Also in the option "Access point with incompatible firmware" the access point was not displayed.

Do you have an idea what I can try next?

[jens-maus]

Do you have an idea what I can try next?

Nothing really. If you read through this issue ticket closely you will find reference to a HomeAssistant related ticket (https://github.com/home-assistant/plugin-multicast/issues/17) which is related to the root cause of this issue: The HmIPW-DRAP and HMIP-HAP rely on UDP multicasting network traffic which is currently not correctly routed to an add-on in HomeAssistant. Thus, the HomeAssistant developers (e.g. @pvizeli) would have to work on this by enhancing their plugin-multicast accordingly. That's the reason why this (no connectivity to a HmIPW-DRAP or HmIP-HAP) is currently a known limitation of using the RaspberryMatic add-on within HomeAssistant.

[firstusing]

@jens-maus thanks for your efforts. I have rebuilt my complete smart home to get the Rasperrymatic installed as Addon within HA. I was reading slowly thru your guideline and had luck to see the issue before I did start. Nothing within my home would work without DRAP and the Second HAP. What can we do to trigger the issue so they will get manpower to the ticket?

Thank you!

[jens-maus]

What can we do to trigger the issue so they will get manpower to the ticket?

If you cannot work on it yourself you can just lean back and wait until someone else is going to implement the missing piece.

[vore]

What can we do to trigger the issue so they will get manpower to the ticket?

Hello! Same here.. I waiting for this in order to switch to Homeassistant. Until I can't use the DRAP I can't change over. Unfortunately this is nothing I could help to implement.

[unaiur]

Warning: multicast and broadcast are not the same!

With multicast, a client has to register to receive the stream of packets. Multicast destination addresses are always in the 224.0.0.0-239.255.255.255 range.

With broadcast, the packet is received by all the hosts in the local network without any extra step. The destination address is the broadcast address of the current subnetwork (for example 192.168.1.255)

After reading this thread, it seems that HMIP uses broadcast, not multicast. So waiting for pmid is pointless.

[unaiur]

Therefore, we just need to use a macvlan docker network in bridge mode as described here: https://docs.docker.com/network/macvlan

Which is the same proposal that @dkadioglu did 8 months ago and all you ignored.

[jens-maus]

Warning: multicast and broadcast are not the same!

Of course not.

After reading this thread, it seems that HMIP uses broadcast, not multicast. So waiting for pmid is pointless.

What exactly from this thread here makes you feel this is actually the case? In fact, I still believe the HAP/DRAP are both relying on UDP multicasting and not broadcasting communication channels and thus haveing pmid running within the HA host should actually solve this issue.

Therefore, we just need to use a macvlan docker network in bridge mode as described here https://docs.docker.com/network/macvlan

Well, this won't help in the HomeAssistant Add-on case, I am afraid since HA does not support to start addons with macvlan. So, yes. the suggestions by @dkadioglu could be added to the documentation so direct docker users could try to get this working the way he was proposing it. But for the HomeAssistant add-on case we need a different solution.

Which is the same proposal that @dkadioglu did 8 months ago and all you ignored.

Well, you can also come up with an own proposal or pull request if you identified a solution for the HAP/DRAP communication issue mentioned here.

[ivo7B4]

Thank you @jens-maus for all the your efforts and your great work, highly appreciated!

I'm still getting into HA, CCU3 and Raspberrymatic as I'm preparing everything for installation.

There's an observation I have where I'm not sure whether this is related to this issue:

• The installation I'm preparing has a CCU3, HAP and 2 DRAPs (and many more devices).
• I've started with CCU3 and have integrated this in HA.
• I've then got into Raspberrymatic. I've installed this on a RaspberryPi4 with a HmIP-RFUSB. After installing the backup, all wireless devices were found quite fast. I struggled however that Raspberrymatic (not as add-on with HA, the normal Raspberrymatic installation) finds the DRAPs with the connected wired devices. I've kept it running for hours. Somtimes after a power reboot of the DRAPs, I was able to detect them. I've tried this multiple times.
• I was considering whether this is a normal issue when applying the backup. I've checked with a second CCU3 where I installed the backup. No issue at all, the DRAPs were found very fast (and also no reboot of the DRAPs was required).

I thus don't know where this issue comes from:

• Is it related to HmIP-RFUSB with RaspberryPI4? Would I not see this issue with Raspberrymatic on a CCU3?
• Something else in my installation? I would not see any other issue however as I can install the backup without any problem on another CCU3.
• Is there a dependency with this issue or is this a separate issue? I share this observation in case there is any relation to this issue (which I don't assume, but I don't know).

[campinge]

Hello all! This has been pending for a while now and I just encountered the same problem. Does anyone know if it can be solved somehow in the meantime?

[jens-maus]

As long as this ticket is open and not closed there is no other solution than not using the HA addon of RaspberryMatic if you want to connect it to a HmIPW-DRAP or HmIP-HAP.

[vore]

Is it done now? Looks good from what I see. Anybody who can test and confirm?

[jens-maus]

Is it done now?

I don't think so. It's done when it's done, thus when this ticket is closed!

[vore]

Is it done now?

I don't think so. It's done when it's done, thus when this ticket is closed!

Hello! The ticket is still open.. it does not look like it is going to be closed. Should I maybe ask on the current status?

[jens-maus]

Hello! The ticket is still open.. it does not look like it is going to be closed. Should I maybe ask on the current status?

No reason to constantly ask for the status here. As already said in october: It's done when it's done. And if there is something relevant to report it will be and if the issue is solved, the ticket will be closed. Thus, no need to further discuss on the status.

[Rainerlan]

I kind of having the same problem: The HA RaspberryMatic-addon does not find the HMIP-HAP - as I read here, because its in another subnet (The LAN-Gateway is found w/o problems however).

I thought of the following solution - But before trying this out, I wanted to check with more experienced people, if the following setup may work:

• As my HA (OS) runs in a Proxmox VM (already), I would add another LXC to install RaspberryMatic as a docker container.
• Configure host mode for the Docker container
• Select the LXC-RaspberryMatic instance in the RaspberryMatic HA integration.

Any thoughts if this might be working? (Don't want to risk my running setup, if it may be evident that this is not going to work as intended)

[jens-maus]

I thought of the following solution - But before trying this out, I wanted to check with more experienced people, if the following setup may work:

• As my HA (OS) runs in a Proxmox VM (already), I would add another LXC to install RaspberryMatic as a docker container.
• Configure host mode for the Docker container
• Select the LXC-RaspberryMatic instance in the RaspberryMatic HA integration.

Any thoughts if this might be working? (Don't want to risk my running setup, if it may be evident that this is not going to work as intended)

This is no discussion or support fora here but a bug/issue tracker. But yes, if you install RaspberryMatic as a dedicated VM (LXC does not work) it will work because then of course no docker is involved which is the root cause of the issue that a DRAP/HAP connection does not work with HA (simply because docker containers can't route multicast UDP traffic).

[marce14-git]

Sorry in advance if my response is not appropriate here: i got it working in a docker container in unraid via a macvlan docker network Has to be running in privileged mode

[jens-maus]

Sorry in advance if my response is not appropriate here: i got it working in a docker container in unraid via a macvlan docker network Has to be running in privileged mode

Sure, there are solutions for other docker environment. But the topic of this issue ticket is more than clear: Get it working in a standard HomeAssistant environment. And there, macvlan is a non avail!

[lukibazuki]

So if I understand this correctly, I'll have to run raspberrymatic on a separate device (not as Add-On in HA), so I can use the HAP as a Gateway. With the HACS Integration "Homematic(IP) Local" I'll still have the possibility to control everything through HA, correct? Is there any workaround to get the HAP working at this moment?

[jens-maus]

Please note, that I was finally able to work on that topic again and now at least for the plain docker container / OCI use case it should be possible to use a HmIP-HAP or HmIPW-DRAP if following my updated documentation at https://github.com/jens-maus/RaspberryMatic/wiki/Installation-Docker-OCI and potentially also using the new install-docker.sh helper script as explained there.

This script (and the docu) utilizes a macvlan docker network setup now like previously explained by @dkadioglu (cf. https://github.com/jens-maus/RaspberryMatic/issues/1373#issuecomment-922508274) and thus should allow the docker container to communicate via UDP multicasting to a HmIP-HAP or HmIPW-DRAP and also make it unnecessary to open dedicated ports because the container will receive a dedicated ip address from the same subnet and thus should act as a "real" RaspberryMatic CCU (also with its own firewall setup).

However, please note, that for the HomeAssistant Add-on use case the issue/limitation discussed here still applies because it is currently not possible to setup the HA add-on docker container to work on dedicated docker networks, thus not being able to use a macvlan docker network. This is still something the HA developers (e.g. @agners, @pvizeli) are working on.