Implement Somfy local API in Overkiz integration

[iMicknl]

The problem

Somfy just announced the Somfy TaHoma Developer Mode, which offers a local API on eligble TaHoma boxes (TaHoma V2, TaHoma switch, TaHoma DIN-Rail v1 & v2).

They also announced that they are sunsetting the Somfy Open API, which is used by the Somfy integration. The API will be supported until June 21st 2022.

The local API is based on the same architecture as the current Overkiz integration in core, thus it would make sense to sunset the Somfy integration and add local support to the Overkiz integration.

Full Announcement

Dear customer,

We are glad to contact you today for the release of a new service we call Developer Mode, available for the different types of TaHoma boxes*.

This new service allows to activate a local API on your TaHoma box and finally connect third-party software to your equipment. A complete documentation of the service is available on GitHub. In a nutshell, the API allows to monitor and control the devices which are paired locally with your TaHoma box without the constraint of going through our servers. Thanks to the new Developer Mode, you can easily integrate your Somfy equipment with third-party home automation softwares.

Support Our support teams are available for any problems or questions regarding the use or operation of the TaHoma system. However, for questions related to the development of third party software using the API, please refer to the page mentioned above for further details. Finally, for problems or questions concerning the use of said software, it is necessary to contact the supplier or developer of the chosen solution directly for assistance.

Warning The applications proposed by Somfy to use its boxes and connected solutions employ a certain number of rules designed to preserve the integrity of your equipment. Please be careful when using the API so as not to damage them. Somfy declines all responsibility for inappropriate use. Similarly, the use of the TaHoma application in conjunction with the API may cause conflicts or contrary actions for which Somfy cannot be held responsible. Also, to ensure the proper functioning of your box, be sure to limit the demands on it.

Sunsetting Somfy Open API It is always hard to say goodbye but Developer Mode is so much better ! Its release therefore announces the depreciation of the Somfy Open API (cloud-based) as the new service will provide a larger number of features (scenarios and group commands being included) but also a larger number of supported devices. Finally, the Developer Mode, by its local nature, avoid any downtime issue or traffic limitation.

If you are a Somfy Open API customer, from the entire Somfy team, we’d like to apologize for any inconvenience this may cause you. We’re committed to making the transition as smooth as possible and that’s why we’ll continue to support this API until June 21st 2022.

Thank you for your understanding and we hope you will enjoy this new service.

*Eligible TaHoma Boxes: TaHoma V2, TaHoma switch, TaHoma DIN-Rail v1 & v2

TODO

• [x] Add deprecation notice (with date / migration path) to Somfy integration
• [ ] Implement local API
  • [x] Support local API in python-overkiz-api
  • [x] Extend Config Flow
  • [ ] Check for self-signed SSL cert docs
  • [ ] Fix diagnostics (client.get_execution_history() does not exist)
  • [ ] Remove auth token when we unload the integration

Integration causing the issue

overkiz, somfy

Link to integration documentation on our website

https://www.home-assistant.io/integrations/overkiz/ https://www.home-assistant.io/integrations/somfy/

[probot-home-assistant[bot]]

overkiz documentation overkiz source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @vlebourl, @tetienne, mind taking a look at this issue as it has been labeled with an integration (overkiz) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[downdrown]

This actually sounds awesome! I have had quite a lot of issues with the cloud API limitations. Is there anything I can help with? I'm really looking forward to this being live in the Overkiz integration! 🚀

[iMicknl]

This actually sounds awesome! I have had quite a lot of issues with the cloud API limitations. Is there anything I can help with? I'm really looking forward to this being live in the Overkiz integration! 🚀

We have a proof of concept working, however we wait until a fix on Somfy's side is released (https://github.com/Somfy-Developer/Somfy-TaHoma-Developer-Mode/issues/4). Until then, we cannot move forward.

[EDelsman]

Grateful for all your effort, but I am getting nervous. The fix this was waiting for seems fixed some time ago, and the old integration is deprecated. Wasn't this supposed to be in the 2022.6 release? This is needed withing three weeks, so it cannot wait till the next major release 2022.7.

[XalaTheShepard]

@EDelsman , if I am not mistaking the depreciation of the Somfy API only affects the Somfy integration and not the Overkiz integration since they both use different login authentifications.

[EDelsman]

Thanks. Let's hope you are indeed not mistaking ;-)

[XalaTheShepard]

It was also mentioned in the 2022.6.0 release notes

[EDelsman]

I read those, but they just said: Switch to Overkiz. So I turned to the documentation, that states: "The Somfy Open API (cloud-based) has been deprecated in favor of the Somfy TaHoma Developer Mode and will shut down after June 21st, 2022. The Overkiz integration will support this new local API and can already be used to control your Somfy devices via the cloud."

So that the first deprecated cloud based is different from the second was not clear to me. I read in it that I should switch to local instead (Which I greatly prefer).

[XalaTheShepard]

They use different logins.

[donparlor]

For those using Somfy MyLink, does this deprecated Somfy integration will affect our HA instance?

[iMicknl]

For those using Somfy MyLink, does this deprecated Somfy integration will affect our HA instance?

Since it is a separate integration, I would assume you won’t face any issues. This will use a different connection than the deprecated ‘Open with Somfy API’.

[donparlor]

Ok great! I guess you are right, I don't feel on a hurry anymore 😂

Have a good day!

[chheiss]

And news here? Local access would be extremely great! I can test, if that helps.

[Mariusthvdb]

so do: https://community.home-assistant.io/t/overkiz-api-and-somfy-api/61448/1829

[rjlee]

Adding a report here as per request in: https://community.home-assistant.io/t/overkiz-api-and-somfy-api/61448/1882

Results from testing the local integration branch following the instructions here: https://community.home-assistant.io/t/overkiz-api-and-somfy-api/61448/1829

My Tahoma Box is: https://www.somfy.me/en-gc/products/2401346/tahoma-box-home-automation and 'Developer Mode' is activated via the Somfy site.

HA version:

Home Assistant 2022.10.4
Frontend 20221010.0 - latest

Observed Behaviour

Autodiscovery of my Tahoma Hub occurs, although multiple instances of the same Hub are detected, although only one has the gateway.local address, others just prompt for the username and password.

Picking the one with the gateway.local address prompts for the username and password with the gateway address prepopulated:

Adding my username and password, this then completes, but the following error is shown in the home-assistant.log and the Overkiz integration is marked as 'Failed to Setup'.

Logger: homeassistant.config_entries
Source: custom_components/overkiz/__init__.py:64
Integration: Overkiz
First occurred: 10:48:16 (1 occurrences)
Last logged: 10:48:16

Error setting up entry gateway-XXXX-XXXX-XXXX.local:8443 for overkiz
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 365, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/config/custom_components/overkiz/__init__.py", line 64, in async_setup_entry
    token = entry.data[CONF_TOKEN]
KeyError: 'token'

[iMicknl]

Thanks @rjlee! I will have a look why this can happen. Are there no other errors in the log? It seems that there is no token generated in the config flow, thus the next steps fail as well. I will see if I can make this more robust in an updated version :).

[rjlee]

No problem, thanks for the hard work. A quick review of the logs doesn't show any additional errors:

grep -i overkiz home-assistant.log
2022-10-19 09:47:25.572 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration overkiz which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2022-10-19 09:48:16.837 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry gateway-XXX-XXXX-XXXX.local:8443 for overkiz
  File "/config/custom_components/overkiz/__init__.py", line 64, in async_setup_entry

[slyoldfox]

@iMicknl

I had the same error locally and added some debug strings.

If I understand correctly, it seems that async_validate_input drops back to the else: statement. The check:

68         if user_input[CONF_HUB] == LOCAL:
69             LOGGER.error("LOCAL")
70             session = async_create_clientsession(self.hass)

Does not match because the value is actually somfy_europe. After changing that check I was able to setup the integration correctly (not sure if it's now actually talking locally) :-)

[rjlee]

I've done a bit more debugging. I've enabled debug and I now see this logline:

CONFIGURING LOCAL INTEGRATION

So, it's activating:

    # Local API vs Cloud API
    # TODO add a helper function to see if local API
    if entry.data.get(CONF_HOST):
        LOGGER.debug("CONFIGURING LOCAL INTEGRATION")
        host = entry.data[CONF_HOST]
        token = entry.data[CONF_TOKEN]


        # Verify SSL blocked by https://github.com/Somfy-Developer/Somfy-TaHoma-Developer-Mode/issues/5
        # Somfy (self-signed) SSL cert uses the wrong common name
        session = async_create_clientsession(hass, verify_ssl=False)

Permalink

Which fails on:

File "/config/custom_components/overkiz/__init__.py", line 67, in async_setup_entry
    token = entry.data[CONF_TOKEN]
KeyError: 'token'

Dumping entry.data gives:

{'host': 'gateway-xxxx-xxxx-xxxx.local:8443', 'username': '[email protected]', 'password': 'mypassword', 'hub': 'somfy_europe'}

From searching the code, I can't see anywhere that CONF_TOKEN gets set though, so I assume that's the issue.

Then, as per https://github.com/home-assistant/core/issues/69558#issuecomment-1284398066

I tried to force local instantiation via changing:

68         if user_input[CONF_HUB] == LOCAL:
69             LOGGER.error("LOCAL")
70             session = async_create_clientsession(self.hass)

To

68         if user_input[CONF_HUB] == 'somfy_europe':
69             LOGGER.error("LOCAL")
70             session = async_create_clientsession(self.hass)

This seemed to get further, but ultimately ended up with an error from pyoverkiz:

Not such token with UUID: aaa-bbb-ccc-ddd

Dumping token and uuid both seem to have valid values FWIW. Checking pyoverkiz shows the latest version is installed in the docker container:

 pip list | grep pyover
pyoverkiz                        1.5.5

At that point, I'm stuck, so hopefully that's helpful.

[slyoldfox]

@rjlee I think I had this at one point as well.

I'm not sure what I did though. Maybe the token is cached on disk and it didn't delete it (I think I saw that skimming through the code?).

Did you delete the integration after changing the line and restart - then readded it?

[rjlee]

After some debugging, I figured it out. The gateway-XXX-XXXX-XXXX.local:8443 doesn't appear to be resolvable on my network, so setting it to the IP address of my Tahama Box 192.168.x.x:8443 and the following change made it work for me:

LOCAL = "somfy_europe"
#LOCAL = "local"
LOCAL_HUB = {
    LOCAL: OverkizServer(
        name="Somfy TaHoma Developer Mode (local API)",
        endpoint="",
        manufacturer="Somfy",
        configuration_url=None,
    )
}

So as per https://github.com/home-assistant/core/issues/69558#issuecomment-1284398066.

@slyoldfox thanks for the help.

@iMicknl FYI - on the issue I experienced and a potential fix.

The only issue that remains is the autodiscovery of multiple instances of the Tahoma hub on the devices page, but that's more an annoyance than a blocker.

Hope that helps.

[slyoldfox]

Ah yah. That was it lol. The exception ended there but the root cause was an Address Could Not Be Resolved exception.

We might need to fix that first. The possible reason is that this is discovered using mdns (?) but my freshly setup ubuntu server doesn't hook into that.

I ended up adding it to my /etc/hosts file:

192.168.2.30 gateway-2017-0514-1111.local

I'm not sure if the hub responds with its IP address somewhere so we can use that instead during the setup?

[iMicknl]

@slyoldfox that is interesting. My preference is to use the mdns address, but the Config Flow will accept the IP address as well for cases where the mdns address cannot be resolved.

[slyoldfox]

Well yes of course the mdns address must be the first option. But if the actual resolving fails during config flow, it would be interesting to have a better handling:

• either fail completely with a decent error and link to documentation (add to hosts file or install a service that resolves it)
• either fall back to IP with a warning message that mdns resolution was not available and it will use the IP instead (that is if that is available somewhere)

[slyoldfox]

For reference this is the stacktrace:

2022-10-20 15:45:00.418 ERROR (MainThread) [custom_components.overkiz] Not such token with UUID: f5a3dcdd-ac10-3e01-6199-5f1124cfffbf
Traceback (most recent call last):
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

Traceback (most recent call last):
  File "/home/hass/.homeassistant/custom_components/overkiz/config_flow.py", line 112, in async_validate_input
    await local_client.login()
  File "/home/hass/homeassistant/lib/python3.10/site-packages/pyoverkiz/client.py", line 159, in login
    await self.register_event_listener()
  File "/home/hass/homeassistant/lib/python3.10/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/pyoverkiz/client.py", line 528, in register_event_listener
    response = await self.__post("events/register")
  File "/home/hass/homeassistant/lib/python3.10/site-packages/pyoverkiz/client.py", line 775, in __post
    async with self.session.post(
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host gateway-2017-0459-8156.local:8443 ssl:default [Temporary failure in name resolution]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hass/.homeassistant/custom_components/overkiz/config_flow.py", line 245, in async_step_local
    user_input = await self.async_validate_input(user_input)
  File "/home/hass/.homeassistant/custom_components/overkiz/config_flow.py", line 114, in async_validate_input
    await client.delete_local_token(gateway_id, uuid)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/pyoverkiz/client.py", line 727, in delete_local_token
    await self.__delete(f"config/{gateway_id}/local/tokens/{uuid}")
  File "/home/hass/homeassistant/lib/python3.10/site-packages/pyoverkiz/client.py", line 793, in __delete
    await self.check_response(response)
  File "/home/hass/homeassistant/lib/python3.10/site-packages/pyoverkiz/client.py", line 864, in check_response
    raise NotSuchTokenException(message)
pyoverkiz.exceptions.NotSuchTokenException: Not such token with UUID: f5a3dcdd-ac10-3e01-6399-5f1124ca0abf

[HpNoTiQ56]

After some debugging, I figured it out. The gateway-XXX-XXXX-XXXX.local:8443 doesn't appear to be resolvable on my network, so setting it to the IP address of my Tahama Box 192.168.x.x:8443 and the following change made it work for me:

LOCAL = "somfy_europe"
#LOCAL = "local"
LOCAL_HUB = {
    LOCAL: OverkizServer(
        name="Somfy TaHoma Developer Mode (local API)",
        endpoint="",
        manufacturer="Somfy",
        configuration_url=None,
    )
}

So as per #69558 (comment).

@slyoldfox thanks for the help.

@iMicknl FYI - on the issue I experienced and a potential fix.

The only issue that remains is the autodiscovery of multiple instances of the Tahoma hub on the devices page, but that's more an annoyance than a blocker.

Hope that helps.

Thank you! I've managed to get it to work instantly by using IP!

[nyroDev]

@slyoldfox that is interesting. My preference is to use the mdns address, but the Config Flow will accept the IP address as well for cases where the mdns address cannot be resolved.

While I tested the local API, i did have the same issue: only IP address worked. In my understanding, it was due to the dev container and the way HA was loaded locally, which makes impossible to reach the host. it would need more testing while others elements will be finished, but IMHO this is not an issue real users will have.

[diedvdyk]

My Tahoma box registers itself in my DHCP as xxxx-xxxx-xxxx where they x's are the serial number of the box, no "gateway" infront of it and no .local after it . After entering xxxx-xxxx-xxxx.mylocaldomain.yy:8443 the local api works super (where mylocaldomain.yy is my local domain) . I hope this helps.

[iMicknl]

@diedvdyk do you have any special networking setup? I don’t think this is default behavior at all..