HmIPW(ired) Device Support - Discussion concerning multi-input-devices

[henningweiler]

Hi,

Currently, I am working on extending pyhomematic to support the Homematic IP Wired components that will come out soon.

The Switch / Relais Devices HmIPW-DRS4 and HmIPW-DRS4were rather easy and I already got them to do what they are supposed to.

Now: Working on the HmIPW-DRI32 proves as a bit more of a conceptual challenge for me and I need your input. The HmIPW-DRI32 is a 32-channel input device... It can handle the following attached devices:

• switches (perma-off, perma-on)
• buttons (short / long press)
• reed contact sensors. In the XML, these three are not distinguishable, although they are configurable in the webUI.

My underlying question however: where to put that device? It is neither an actor, nor a sensor ?! My gut feeling was to place it in sensors, but wanted to check-in with you before, @danielperna84. I'm also asking, because I could not explicitly find the input channels of e.g. the HMW-IO-12-Sw7-DR

Cheers!

[danielperna84]

I think the HMW-IO-12-Sw14-DR is similar. It detects how the channels are configured (I'm sure you can get the configuration from the MASTER paramset somehow) and dynamically adjusts the channels. In this case they can also be configured in different ways. However, not all cases can be replicated. Some channel configurations will just not work.

The HMW-IO-12-Sw7-DR works because it has the HelperEventRemote helper. When subclassing that helper the events will be passed on to the Home Assistant event bus. That's how buttons (short/long press) would be implemented if I recall correctly.

[henningweiler]

Thanks!

So, I managed to add the HmIPW-DRI16/32 devices as an actor.

All fun and games, until I want to check the channel mode and feed that intel to Homeassistant.

The function getParamset() seems to have issues on HMIP devices ... at least, I am not able to call this function on the proxy for any of my HMIP(W) devices without getting a FAULT form the RPC. It is OK for HM(W) devices! Calling getParamset() from outside of HomeAssistant in a simple script works, as long as Homeassistant is not connected to the CCU3.

The code:

    def __init__(self, device_description, proxy, resolveparamsets=False):
        super().__init__(device_description, proxy, resolveparamsets)
        self._hmipw_keys = []
        self._hmipw_binaries = []

        for chan in self.ELEMENT:
            address_channel = "%s:%i" % (self._ADDRESS, chan)

            try:
                channel_paramset = self._proxy.getParamset(address_channel, "MASTER", 0)
                channel_operation_mode = channel_paramset.get("CHANNEL_OPERATION_MODE") if "CHANNEL_OPERATION_MODE" in channel_paramset else 1
                if channel_operation_mode == 1:
                    LOG.debug("IPWInputDevice: Added %s as ACTION KEY", address_channel)
                    self._hmipw_keys.append(chan)
                elif channel_operation_mode in [2, 3]:
                    LOG.debug("IPWInputDevice: Added %s as BINARY_SENSOR", address_channel)
                    self._hmipw_binaries.append(chan)
            except Exception as err:
                LOG.error("IPWInputDevice: Failure to determine input channel operations mode of HmIPW input device %s: %s", address_channel, err)

        self.ACTIONNODE.update({"PRESS_SHORT": self._hmipw_keys,
                                "PRESS_LONG": self._hmipw_keys})
        self.BINARYNODE.update({"STATE": self._hmipw_binaries})

    @property
    def ELEMENT(self):
        """ General channel definition """
        if "HmIPW-DRI16" in self.TYPE:
            return [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
        elif "HmIPW-DRI32" in self.TYPE:
            return [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
        return [1]

The error in home-assistant.log for each channel:

2019-02-14 22:53:43 ERROR (Thread-19) [pyhomematic.devicetypes.actors] IPWInputDevice: Failure to determine input channel operations mode of HmIPW input device 001234567:01: <Fault -1: 'Generic error'>

No log is shown on the CCU3. No additional information is given by any of the platforms.

Any ideas? Even ideas on how to debug this further are greatly appreciated.

Cheers!

[danielperna84]

It is OK for HM(W) devices!

So you can confirm, that the flow generally is working? I did not implement this on my own since I don't have any wired devices, and I don't knoe how thoroughly it has been checked. So I don't know if it has been broken all the time.

Regarding you observations: In this issue for the HmIP-MOD-RC8 the same problem seems to be present. Apparently there's some problem to use the same approch for HmIP devices. I don't know what a reasonable solution for this would be. My guess is, that during startup there are too many requests, and the CCU can't handle them all at once. That's why the proxy object already uses locking. But for HmIP devices this seems to not be enough.

As a general improvement that could help to put less stress on the CCU the DEVICEFILE could be implemented in Home Assistant. Maybe also a general caching for values similar to what has been done here might help. The key seems to be to find out a method to resolve the paramsets programatically, which does work when done manually. And nobody knows why it does not work when done from pyhomematic. 😞

I did a little googeling, and it seems as if this problem has been around for some time (04/2018): https://community.openhab.org/t/recuring-restapi-errors-for-homematic-ip-binding-and-hmip-etrv/44212

And if you really want to dig deep: Here's a strace-binary that at least works on my RaspberryMatic (you can save it to /usr/llocal). All the common ports (2001, 2010 etc.) first go through a lighttpd, but probably get redirected somewhere else later on. I'm sure there are protocol-specific services. If you attach to one of those with strace and see what happens internally, you might find out what is causing this issue and maybe even can file a bug report for the CCU.

Edit: If you do netstat -tlnp | grep 32010 you'll get a Java Process that does show some activity when manually firing a XML-RPC request at 2010. Edit: It seems like the Java Process is looking up the data from /etc/config/crRFD/data. The files there seem to be serialized with kyro. Since I don't have that error I don't know what step fails in your case.

[henningweiler]

Soo ... I did some digging ...

0) found out how the port-routing works and who is the master process:

# cat /etc/lighttpd/conf.d/proxy.conf
[...]
#-------------------------------------------
# crRFD Legacy XmlRpc TLS (:42010 -> :32010)
#-------------------------------------------
$SERVER["socket"] == ":42010" {
  include "/etc/lighttpd/conf.d/sslsettings.conf"
  proxy.server = ( "" => ( "localhost" => ("host" => "127.0.0.1", "port" => 32010)))
  include "/var/lighttpd_auth.conf"
}
$SERVER["socket"] == "[::]:42010" {
  server.use-ipv6 = "enable"
  include "/etc/lighttpd/conf.d/sslsettings.conf"
  proxy.server = ( "" => ( "localhost" => ("host" => "127.0.0.1", "port" => 32010)))
  include "/var/lighttpd_auth.conf"
}
[...]

And netstat -tlnp | grep 32010 in combination with ps -ef reveals that 32010 is keyed to our hm main thread:

  624 root     28:43 java -Xmx128m -Dos.arch=arm -Dlog4j.configuration=file:///etc/config/log4j.xml -Dfile.encoding=ISO-8859-1 -Dgnu.io.rxtx.SerialPorts=/dev/mmd_hmip -jar /opt/HMServer/HMIPServer.jar /etc/crRFD.conf /etc

1) cranked up the log levels of the ccu in /etc/config/log4j.xml to DEBUG ==> Nothing to be found in the logs concerning our Generic Fault -1 or anything else where any of the logs would indicate any reaction to the getParamset.

2) connected strace to the main java process ==> Also no luck. It would make sense to have strace actively running, if a process or thread would crash to see something. But nothing crashes, so strace ain't helpful here, imho

Preliminary conclusion Seems to have something todo with the overall pyhm <-> ccu communication and/or some timing issues. Will have an eye out for more intel on this and will dig deeper in the rpc server to maybe find something there.

Side notes Observations alongside the debugging endeavours:

• Restart the CCU and use a variation of example.py to call getParamset works flawlessly
• Let the CCU running, connect pyhomematic from homeassistant --> both homeassistant as well as the example.py variation will fail (gerneric Fail -1 after some long seconds of waiting)
• Stop the homeassistant server and try the example.py to call getParamset will fail again (gerneric Fail -1 after some long seconds of waiting)
• During that time, the CCU appears to have slowed down. However, there is no impact on CPU / RAM consumption or anything measurable outside of the JVM on os/hardware level

[danielperna84]

2) connected strace to the main java process ==> Also no luck. It would make sense to have strace actively running, if a process or thread would crash to see something. But nothing crashes, so strace ain't helpful here, imho

How did you work with strace? I can generate a Generic error when manually running a pyhomematic session without opening the firewall on my windows machine. The CCU therefore isn't able to send the answer for listDevices. It's not the same error, but it generates an error. And I can see that in strace, which tells me that a connection has timed out (this shows up in the hmserver.log as well). The command I use is ./strace -f -s 256 -o trace.out -p $thejavapid. I just leave that running, and from another terminal start the pyhomematic session. Afterwards I search the trace.out file for Generic. And not too far above from that I see the timeout messages.

[henningweiler]

Thanks! Tried again yesterday:

Method:

1. started it via ./strace -f -s 256 -o /tmp/trace.out -p $thejavapid
2. left it on for about 15min
3. started the requests from both homeassistant and the example.py

Observations:

• strace connected to a total of 53 sub processes
• trace.out grew to measure 15MB
• There are a LOT of timeouts present, even if no action / connection is happening to the CCU3:

# egrep -i timeout /tmp/trace.out | wc -l
7680

• The Generic Error is thrown once upon calling getParamset from homeassistant:

# egrep -i generic /tmp/trace.out
681   writev(95, [{"HTTP/1.0 200 OK\r\nContent-Length: 257\r\n\r\n", 40}, {"<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?><methodResponse><fault><value><struct><member><name>faultCode</name><value><i4>-1</i4></value></member><member><name>faultString</name><value>Generic error</value></member></struct></value></fault></methodResponse"..., 257}], 2) = 297

• Checking PID 681 in this file shows a timout in the futex call:

# egrep -i "^681 " /tmp/trace.out | egrep -i generic -A10 -B10
681   epoll_wait(28, {{EPOLLIN, {u32=26, u64=7606579873707982874}}, {EPOLLIN, {u32=95, u64=7539843726969929823}}}, 4096, 38182352) = 2
681   read(26, "\1", 128)               = 1
681   read(95, "POST /RPC2 HTTP/1.0\r\nHost: 10.0.4.40:42010\r\nAuthorization: Basic aGFfdXNlcjpBbEtwUVdJYUphcXpFejFhYXhJeWxYdXBNV3lyVEZDWUF0SXJrUkhyTDdSdFBnRW1rbVVTVVlTcXBBWmVjMUNU\r\nAccept-Encoding: gzip\r\nContent-Type: text/xml\r\nUser-Agent: Python-xmlrpc/3.5\r\nContent-Length:"..., 1024) = 669
681   futex(0x75281a6c, FUTEX_WAKE_PRIVATE, 1) = 1
681   futex(0x68d29e94, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
681   <... futex resumed> )             = 1
681   epoll_wait(28,  <unfinished ...>
681   <... epoll_wait resumed> {{EPOLLIN, {u32=26, u64=7606579873707982874}}}, 4096, 29999) = 1
681   read(26, "\1", 128)               = 1
681   write(27, "\1", 1)                = 1
681   writev(95, [{"HTTP/1.0 200 OK\r\nContent-Length: 257\r\n\r\n", 40}, {"<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?><methodResponse><fault><value><struct><member><name>faultCode</name><value><i4>-1</i4></value></member><member><name>faultString</name><value>Generic error</value></member></struct></value></fault></methodResponse"..., 257}], 2) = 297
681   getsockopt(95, SOL_SOCKET, SO_LINGER, {onoff=0, linger=0}, [8]) = 0
681   dup2(12, 95)                      = 95
681   epoll_ctl(28, EPOLL_CTL_DEL, 95, 68a2e7c0) = -1 ENOENT (No such file or directory)
681   close(95)                         = 0
681   epoll_wait(28, {{EPOLLIN, {u32=26, u64=7606579873707982874}}}, 4096, 38162337) = 1
681   read(26, "\1", 128)               = 1
681   epoll_wait(28,  <unfinished ...>
681   <... epoll_wait resumed> {{EPOLLIN, {u32=26, u64=7606579873707982874}}}, 4096, 38162336) = 1
681   read(26,  <unfinished ...>
681   <... read resumed> "\1", 128)     = 1

Result This strengthens the idea that something times out somewhere. However, I cannot tell how or why. Will try again w/ the new original firmware version as well as the new RaspMatic, once it is available.

Ideas welcome ;)

Stay tuned 😇

[danielperna84]

Hhmm, so this is something within the Java Process. Maybe thread-related and doesn't work well with parallel work.

[henningweiler]

This is all very odd. Maybe @jens-maus from RaspberryMatic got some idea here?

In a nutshell: calling getParamset fails on the CCU side, when called from a Thread spawned off of pyhomematic / home assistant. Unfortunately, the CCU is insufficiently verbose to actually tell us what is going on. Maybe you, Jens, got an idea on how to dig this up further?

Meanwhile, I'll go on and try to get a lead somewhere.

[danielperna84]

Would you mind creating a PR just for the HmIPW-DRS4 and HmIPW-DRS4? You said you got them working already, and support for 2 devices is better than none at all. 😉

[henningweiler]

Absolutely. As soon as I have moved to the new home, I can finalise the PR and send it your way : )

[danielperna84]

Hey, a few months have passed now, so I wanted to bring this issue back to your attention. I had a look at your pyhomematic fork, but you haven't pushed anything to it yet. For now we still don't have any support for HmIPW. Would be nice to hear back from you. 👍

[steckenpferd]

Anyone?

[p0l0]

Is anyone working on this issue? To avoid multiple developments, it would be great to coordinate.

I will receive my HmIPW Actors in the next 2-3 weeks, and if no one is working on it, i would start trying to implementing it.

I will have following HmIPW Actors:

• HmIPW-DRBL4
• HmIPW-DRD3
• HmIPW-DRI32
• HmIPW-DRS8,
• HmIPW-FIO6

[danielperna84]

@p0l0 I guess no one will mind if you give it a shot. 👍

[henningweiler]

Hey,

Yeah, that didn't go quiet as planned ... sorry about that.

As a reference, I will push my take on DRI16/32 as well as the DRS4/8

The logic implemented by me appears to work now with HM/CCU3 Firmware >= 3.49.17

@p0l0, you can build on top of this (once pushed), if you feel like it.

Caveat: Pls note that I did not push the related changes to the homeassistant addon to incorporate the new sensors/switches.

[p0l0]

@henningweiler it would be great that you push your changes.

I just received my Actors and I will start installing and configuring everything as soon as possible, so that I probably will be able to start with the development this weekend.

[henningweiler]

@p0l0 things are merged to devel now. Good luck!

[p0l0]

I have implemented successfully the HmIPW-DRD3, HmIPW-SPI and HmIPW-DRBL4 actor.

Unfortunately for HmIPW-DRI32 and HmIPW-FIO6, I have the same problem with "Generic error".

I'm using a Raspberry Pi 3B with RPI-RF-MOD and piVCCU, using an SD Card. What I noticed, on startup of HomeAssistant, is that the WebUI and XML-RPC is not responding until the whole HomeAssistant initialisation has finished.

This is only my current Test setup, I will change it in the next weeks to run on an USB SSD and will use raspbian+debmatic. Maybe the failure is due to lack of ressources.

@danielperna84 if you want I can do the pull request for HmIPW-DRD3, HmIPW-SPI and HmIPW-DRBL4, including the changes in the Homeassitant Integration.

[danielperna84]

Unfortunately for HmIPW-DRI32 and HmIPW-FIO6, I have the same problem with "Generic error".

Is you version of piVCCU recent enough to be comparable to the 3.49.17 CCU firmware? According to @henningweiler's statement here, a newer firmware resolves the issue.

@danielperna84 if you want I can do the pull request for HmIPW-DRD3, HmIPW-SPI and HmIPW-DRBL4, including the changes in the Homeassitant Integration.

Sure, go ahead. 👍

[p0l0]

My piVCCU is version is the latest, 3.51.6. I have the feeling, that it's a ressource problem, because nothing works in the WebUI and XML-RPC (using Postman) until it finishes. I will give it a try with the new setup as soon as it's ready.

[p0l0]

@danielperna84 should I wait until new pyhomematic release is created before making the pull request for the Home Assistant integration or can I do it already without updating the dependency version?

[danielperna84]

Is there more to it than just adding the new classes in Home Assistant? I usually do that when I do the update for pyhomematic in Home Assistant.

[p0l0]

No, just added the classes:

https://github.com/p0l0/home-assistant/commit/5fc29542ea02cf199b3c795637fad5a1bcb823e9

[danielperna84]

I have copied the diff to your PR here. When releasing the new version I'll see that and replicate those changes in Home Assistant. 👍

[p0l0]

The error with "Generic error" responses is now solved, it was a lack of system resources. With USB Boot and debmatic (CCU Version - 3.51.6.43), everything is working like expected. I will finish the implementation of HmIPW-FIO6 and then push the changes, including the change for MotionDetection to be a sensor instead of actor.

[p0l0]

@danielperna84 Unfortunately the "Generic error" is back. I implemented HmIPW-FIO6, but due to the "Generic error", I only was able to test the output Channels, the input channels should work, but aren't tested.

I commited the changes to my fork, if you want I can make a merge request, but Input Channels for HmIPW-FIO6, DRI16 and DRI32 are not usable as long as CCU is returning the "Generic error".

I'm trying to find out were the error comes from, but as I'm not using a "real" CCU3, there is also no support from EQ-3 and they are not accepting the issue request.

[danielperna84]

Maybe the stability will improve once there is progress with this issue. Even though it seems unrelated, maybe solving the buffering problem could fix other parts of the API. 🤷‍♂️

[timniemeier]

@p0l0 – were you able to solve the issue?

I also have DRI32 but cannot get it up and running. It is part in the XML from HM, but does not show in Lovelace.

[p0l0]

@timniemeier I'm will be move into our new House in the next 2-3 weeks, for that I'm setting up everything (HA, CCU, HmipWired). I will give an update as soon, as I have finished with the setup.

[TobiasBurg]

any update @p0l0 ? i have the same status like @timniemeier

So if someone need assistance i can help :)