homeassistant: update to v2022.10.5

[hgy59]

Description

Since we can now build Cryptography wheel with rust, it is time to update homeassistant.

Fixes #

Checklist

• [ ] Build rule all-supported completed successfully
• [ ] New installation of package completed successfully
• [ ] Package upgrade completed successfully (Manually install the package again)
• [ ] Package functionality was tested
• [ ] Any needed documentation is updated/created

Type of change

• [x] Package update
• [ ] This change requires a documentation update (e.g. Wiki)

[hgy59]

@th0ma7 homeassistant depends now on orjson wheel.

The build of orjson wheel needs manturin instead of setuptools-rust to build with rust. I have fixed this by installing maturin with pip in native/python310 and into the cross build env.

In this context I found, that spk/homeassistant/Makefile did not add the path to native/python310 bin folder to the environment (and could not find maturin...).

I suppose that all packages creating wheels must set the path for native/python to be found first, otherwise it uses python3 (and wheel, pip, ...) of the dev environment that is still on python 3.9.

[th0ma7]

That is really really odd. A few things:

• When creating the crossenv directory cross/python310 does use proper pip from PIP_NATIVE variable which in turns points to native/python310
• in theory, neither manturin nor setuptools-rust are needed into native/python310. They are only needed part of the work-<arch>-<version>/crossenv being generated duging cross/python310
• As you are invoking the work-<arch>-<version>/crossenv/cross/bin/* cross-environment binaries when cross-compiling wheels, in turns you make use of native/python310 libraries.
• even if you build pure-python wheel, you'll still be calling native/python310

So I wonder how did you ended-up calling python3.9, how was that even possible?

[Safihre]

Does Homeassistant really need orjson? SABnzbd can also use it for example, but will just fallback to regular json if not available. Does the package not start without it?

[hgy59]

Does Homeassistant really need orjson? SABnzbd can also use it for example, but will just fallback to regular json if not available. Does the package not start without it?

@Safihre I didn't try to start it without orjson, as this is listed in the dependencies in homeassistants pyproject.toml

[ixje]

Does Homeassistant really need orjson? SABnzbd can also use it for example, but will just fallback to regular json if not available. Does the package not start without it?

fwiw I looked at it. It's part of their helpers/utils and there is no logic to fallback to the default. Their utils use specific options when dumping such as orjson.OPT_PASSTHROUGH_DATACLASS with the note

orjson supports serializing dataclasses natively which eliminates the need to implement as_dict in many places when the data is already in a dataclass. This works well as long as all the data in the dataclass can also be serialized.

Which makes me believe they rely on it later in their code base. Finally, the home_kit component also has orjson as requirement.

[hgy59]

So I will drop ARMv5 support of homeassistant, as maturin wheel cannot be built for it.

Affected models with DSM 6 are

• DS213air, DS213, DS413j
• DS112, DS112+, DS212, DS212+, RS212, RS812, DS212j, DS112j
• DS111, DS211, DS211+, DS411slim, DS411, RS411, DS211j, DS411j

EDIT: ARMv5 support was already dropped by numpy.

[Safihre]

Ancient models. So just put it in release notes if anyone is wondering later.

[ixje]

🎉 can't wait for this release. Thanks for the work 👍

[hgy59]

@ixje the default configurations are working now.

If you wanna try, wait for the github action to complete (successfully I hope). But be aware, that intergrations might need a manual installation

To manually install missing python modules (wheels):

1. ssh into your diskstation
2. navigate (cd) to folder /var/packages/homeassistant/target/env/bin
3. call ./pip install --no-deps "python-package==version"
4. report working integrations and/or missing python modules here

you find the spk packages to download after gh action completes under "checks - build - summary - artifacts" to find the matching package see our wiki under Package Archs

Thanks.

[ixje]

I won’t be home until monday, but will check then and let you know. Thanks!

On 17 Nov 2022, at 18:47, hgy59 @.***> wrote:

 @ixje the default configurations are working now.

If you wanna try, wait for the github action to complete (successfully I hope). But be aware, that intergrations might need a manual installation

To manually install missing python modules (wheels):

ssh into your diskstation navigate (cd) to folder /var/packages/homeassistant/target/env/bin call ./pip install --no-deps "python-package==version" report working integrations and/or missing python modules here you find the spk packages to download after gh action completes under "checks - build - summary - artifacts" to find the matching package see our wiki under Package Archs

Thanks.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[Safihre]

Working hard @hgy59! 👍 👍

[hgy59]

Hi homeassistant users! I currently have some questions on how to resolve the huge Dependency Stuff

When I released the last ha package (2021.9.7-15) I started to bundle only the cross compiled wheels into the package and load all the pure python dependencies at installation time (this takes up to 60 minutes to install HA on an old weak model like DS-115j). One reason to do so, was to keep the spk files as small as possible.

But it is a time consuming task to find all the dependencies (I created a test client that tries to add all integrations and added the modules until respective error logs where gone).

But I actually do not want to include a list of all the (pure) python modules anymore and install all those packages at installation time. This can be done for the basic requirements to run homeassistant at its own. But not for the huge (constantly growing) number of integrations. I am (as a non python programmer) wondering, why some python modules in homeassistant (or any other python program?) are installed on the fly and others fail with No module named ....

• Isn't there an automatic dependency resolver in Python available, that installs missing dependencies on the fly (without adjusting the code of homeassistant)? (I found pipimport, but AFAICT you have to use it inside python code)
• What kind of application/tool could provide a solution for automatic python module installation?
• Exists this dependency problem only in Home Assistent Core? (don't know the other distributions [supervisor?], but those do not run on Synology DSM anyway).

So far a manuall installation of missing python modules is the only solution but on spk package updates, all those manual installations are lost.

some early ideas:

• create an additional requirements file that can be manually updated by users and that is used to install additional dependencies at service start (and that will survive package updates)
• create a tool that watches the homeassistant log files and calls pip to install missing modules on the fly
• ask homeassistant developers to improve the code to automatically (or interactivly) install missing dependencies when adding integrations

...probably I am too far away from the python mystery and there exists a simple solution?

[ixje]

I am (as a non python programmer) wondering, why some python modules in homeassistant (or any other python program?) are installed on the fly and others fail with No module named ...

Do they really install on the fly? I’m without computer atm so can’t look at the code how it might work. It seems logical to have a base set of requirements you install upon first installation and then some optional ones depending on what hw support you need. The normal way (not homeassistant specific) to handle this would be to add the optionals to “extras” of the installation config (setup.py,setup.cfg or pyproject.toml, whatever flavour a project chooses) which you would then install via e.g “pip install homeassistant[<optional_package>]”. I’m a HA novice user but I imagine they have a friendlier solution than redirecting users to the console. It’s not something I’ve encountered before so can’t really add much there. Tbh the core project seems messy with the duplication of requirements in both pyproject.toml and the various requirements.txt files.

Isn't there an automatic dependency resolver in Python available, that installs missing dependencies on the fly (without adjusting the code of homeassistant)? (I found pipimport, but AFAICT you have to use it inside python code As a Python programmer I’ve never seen this. As said above the standard approach is making “extras” available or just installing every possible dependency. Runtime resolving is also not straight forward todo as the import name in the code does not have to be the same as the name required to install through pip. What kind of application/tool could provide a solution for automatic python module installation? Exists this dependency problem only in Home Assistent Core? (don't know the other distributions [supervisor?], but those do not run on Synology DSM anyway). I’d be interested to learn how they handle the dependencies of optional integrations. I suspect they’ll just do a “pip install” via shell execution using Subprocess, but then again if you say you’re getting “Module not found”, then it doesn’t sound like they have a great solution for it. So far a manuall installation of missing python modules is the only solution but on spk package updates, all those manual installations are lost.

some early ideas:

create an additional requirements file that can be manually updated by users and that is used to install additional dependencies at service start (and that will survive package updates) create a tool that watches the homeassistant log files and calls pip to install missing modules on the fly ask homeassistant developers to improve the code to automatically (or interactivly) install missing dependencies when adding integrations I don’t know anything about how spk package updates work so what I’m about to say might not work; you can call “pip freeze” from a shell to get the list of currently installed dependencies + versions. If you can execute this prior to update installation you can fetch the list of all requirements they installed and reinstall those after the update.

[ixje]

Got delayed a bit, just finished trying to install. I installed the wheels from package/share/wheelhouse/ all fine. Then tried to install /package/share/postinst_components_requirements.txt. I solved about 5 conflicts then stopped because I'm wondering if it is worth it. Where did this list come from and is it up to date?

[daxcore]

I had to install python module "natsort" by my own to get work one of the "helpers" - I guess it was the "utility meter" or "Riemann sum integral" helper.

[abwebc]

Do we know when this could be merged?

[chickenandpork]

Is there an updated build container for this? I would normally use the latest container in GHCR but that lacks the rust toolchain.

My usual go-to looks like this:

docker run -it -v ${HOME}/src/spksrc:/spksrc -w /spksrc \
    ghcr.io/synocommunity/spksrc \
    /usr/bin/make homeassistant-clean spk-homeassistant-denverton-7.0

I'm a python guy, but not a rust guy outside of limited cmake-based rust builds

I wrapped the build container with a quick rust-equivalent of get-pip like this, but it failed:

FROM ghcr.io/synocommunity/spksrc:latest
  
RUN curl -o installer --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs && sh ./installer -y

(that's a curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh per documentation, but a temp file to allow adding -y param to run non-interactively)

Failed as:

Building wheels for collected packages: maturin
  Building wheel for maturin (pyproject.toml): started
  Building wheel for maturin (pyproject.toml): finished with status 'error'
...
      writing manifest file 'maturin.egg-info/SOURCES.txt'
      running build_ext
      running build_rust
      error: can't find Rust compiler

... so there's something else missing in the rust toolchain.

@hgy59 if you had to update the build container, can you document it, or have I made a silly error in my repro of building this SPK ?

[chickenandpork]

@hgy59 @ixje

I am (as a non python programmer) wondering, why some python modules in homeassistant (or any other python program?) are installed on the fly and others fail with No module named ....

• Isn't there an automatic dependency resolver in Python available, that installs missing dependencies on the fly (without adjusting the code of homeassistant)? (I found pipimport, but AFAICT you have to use it inside python code)
• What kind of application/tool could provide a solution for automatic python module installation?
• Exists this dependency problem only in Home Assistent Core? (don't know the other distributions [supervisor?], but those do not run on Synology DSM anyway). So far a manuall installation of missing python modules is the only solution but on spk package updates, all those manual installations are lost.

I think it is as you say: there is a basic set of dependencies installed, and dependencies of integration are installed as needed. For example, take a look at https://github.com/home-assistant/core/blob/dev/pyproject.toml#L26 ... this might fail on case-by-case with integrations that don't describe dependencies, assuming all servers have certain modules at certain versions. I think it may be important to see which fail specifically.

(@hgy59 pyproject.toml is the newer way of recording the metadata about a python thing; IIUC it should replace setup.py) You'll see just a basic set of requirements; based on how HASS used to act, others were installed as-needed based on the integration I was activating, but I don't have hard proof of that. I know that python won't discover a missing dependency until you interpret the affected file, so there might be some scaffolding around the activation of a new integration to stage in the dependencies.

Your list should probably be an automatic pip freeze or pip-compile of the dependencies based on this file -- every install of this SPK should start out the same to avoid identical installs reaching different results. I realize I'm biased: I'm a bazel guy, and I use version-locked python dependencies for aggressive consistency. I use the bazel stuff for pip-compile so I never run it myself, so I'm rusty there too :(

The idea of an extra list of dependencies, or archives to lay into the config/custom_components that would survive updates would be awesome!

[davesc63]

Testing out your latest build. I am getting python path issues and packages not installing.

Main errors are: Unable to install package packageName==0.0.15: ERROR: Will not install to the user site because it will lack sys.path precedence to "dependancy" in /volume1/@appstore/homeassistant/env/lib/python3.10/site-packages

I can resolve that issue manually, but it makes almost every integration receive an HTTP/500 error in the UI.

Another issue is - Cannot reload configuration.yaml using Developer Tools - YAML - Restart Need to stop HA via DSM Package center

[davesc63]

Ok - I think I have resolved my python / installation issues.

Issue: No integrations were installing their dependencies.

Error messages: When initiating the installation of an integration from the GUI: Defaulting to user installation because normal site-packages is not writeable

Then in the HA log: Unable to install package packageName==0.0.15: ERROR: Will not install to the user site because it will lack sys.path precedence to "dependancy" in /volume1/@appstore/homeassistant/env/lib/python3.10/site-packages

I noticed that /volume1/@appstore/homeassistant/env/lib/python3.10/site-packages has "root" as the owner

Home Assistant user is "sc-homeassistant"

Resolution: I ran sudo chown -R sc-homeassistant /volume1/@appstore/homeassistant/env/lib/python3.10/site-packages

I was then able to add integrations from the GUI and they would resolve/install their dependencies as required.

Two packages I was having issues adding:

• Enphase Envoy
• Aussie Broadband

Both integrations would return a "config flow" error when attempting to add via the HA GUI, then looking in the logs would give the error around the python user-site

[hgy59]

@davesc63 thank you for looking into this and finding a solution. I will fix the installer.

[hgy59]

@davesc63 installation issue is fixed now (was for DSM < 7)

[davesc63]

@hgy59 Great work!

I still cannot restart homeassistant from Developer Tools - YAML - Restart

It hangs, DSM shows homeassistant as stopped and I need to start it from DSM. Haven't got much more info yet

[hgy59]

It hangs, DSM shows homeassistant as stopped and I need to start it from DSM. Haven't got much more info yet

HA Core has changed. It does not have an option to create a pid-file anymore and you can only start and stop it within the DSM Package Center.

[hgy59]

I had to install python module "natsort" by my own to get work one of the "helpers" - I guess it was the "utility meter" or "Riemann sum integral" helper.

After trying to add all integrations, natsort==8.2.0 was installed on the fly.

---

Now almost all integrations that can be configured in the Web UI are working (see the updated wiki page

The follwing integrationes do not work

• Google Nest
• IKEA TRÅDFRI
• SMS notifications via GSM-modem
• Tasmota (may run, can't verify without a compatible device)
• WiLight

IKEA TRÅDFRI runs with the current SynoCommunity Package (v2021.9.7-15) Is this a blocker for releasing this package as is?

[davesc63]

Ship it! :)

[hgy59]

@th0ma7 the github action fails to build, but my local builds succeed. It seems to be related to the "setup of rust" as the errors are:

ERROR: Could not build wheels for maturin, which is required to install pyproject.toml-based projects

Probably related to #5508 ?

EDIT: And #5570 fails to build deluge with:

$ make arch-x64-6.1 -C ./spk/deluge
      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs)/ is the recommended way to download and update the Rust compiler toolchain.
      [end of output]

[chickenandpork]

@th0ma7 the github action fails to build, but my local builds succeed. It seems to be related to the "setup of rust" as the errors are:

ERROR: Could not build wheels for maturin, which is required to install pyproject.toml-based projects

Probably related to #5508 ?

was rustc added to the standard build image? When I tried a while back it wasn’t.

[hgy59]

was rustc added to the standard build image? When I tried a while back it wasn’t.

In #5508 the rust toolchains where removed from the image and should be installed on the fly. I have the same image and it works locally, but probably rustc is installed already in the distrib folder. It works with the distrib folder mounted to a local volume (containing distrib/cargo/bin/rustc), but it might not work wirth an empty distrib folder when running from scratch.

[th0ma7]

The rust toolchain is now downloaded per arch based on spksrc.tc.mk.

I just tested it 2 weeks ago for py310-311 upgrade pr and all was working all right. Perhaps this is only a temporary github networking issue or due to a new commit since then?