Improve Headless Support on Linux (GUI Dependencies)

[lynix]

Feature request type

I'm running a home automation server (actually the same Linux machine I'm using as NAS) to which I'd like to connect my ConnBee for reading sensors.

I find it very odd that deconz has only one binary with Qt among its dependencies. I know that it can be run in headless mode via deconz.service, but the package itself pulls in so many X11 (and Wayland) related packages which is stupid for a headless (server) system.

Description

My suggested solution to this would be to split up deconz into multiple binaries (or even packages) so that you can have it running as daemon without installing all the dependencies for the GUI.

I'm not sure how much of QtNetwork and the-like you use in the headless code path of the binary, so feel free to discard my suggestion if headless operation still requires much of it.

Considered alternatives

The only alternative (which I'm currently taking) is setting up a Debian/Ubuntu container that has all the X11 packages, just for running deconz. That's quite overkill in my eyes, but at least it keeps the 'host system' (NAS) minimal.

Additional context

I'd like to use this section to thank you for all the work and effort you put into your Zigbee related projects! I really appreciate that you have chosen an open development style that allows synergies with the community.

[Mimiix]

@manup

[manup]

I absolutely agree that the GUI and headless parts need to be separated into two packages. This is actually already an ongoing process internally for quite some time. deCONZ in it's early years wasn't developed with headless in mind and now has many intermingled classes which can't be separated easily. This refactoring is happening but at a slow pace with no ETA.

Qt itself will stay as a dependency, but the X11/Wayland related parts won't be needed for headless setup.

One long term goal is to expose the GUI node parts via REST-API so that even for headless installations a remote client can display interactive nodes, with all features of deCONZ GUI, for example in a browser.

The suggested alternative is in fact the only workaround to keep the host system X11 free until the slim headless version arrives.

[lynix]

I know this is slightly off-topic here but I'd like to give a quick report back on my attempts of working around this GUI dependency using containerization.

I am aware of the fact that there is an official Docker image available, but to be honest I prefer LXC or plain systemd-nspawn containers over Docker and already had other LXC containers on my NAS so I tried to go with that. After all, these technologies rely on the same in-Kernel isolation mechanisms so they should give comparable results - or at least so I thought.

I set up a Ubuntu 18.04 container and strictly followed the ConnBee Install Guide regarding installation of deCONZ. Then I had to jump several hoops to get the application running:

• forward the USB device to the LXC container (lxc.cgroup.devices.allow and lxc.mount.entry for the device node)
• sync GIDs so the udev rules of the host make the device accessible to the plugdev group inside the container
• remove capabilities from the systemd unit as I did not want the binary to have them (in my configuration they are not required anyway) and LXC blocked them (fixable using lxc.cap.keep)

(OT: I see room for improvement regarding packaging there. Instead of relying on hard-coded UID 1000 it would be much better to have a dedicated user created during package installation, preferably with UID<1000 and its home directory below /var/lib to match the way many other daemons are set up. Would happily send pull requests for that.)

The application then seemed to work, i.e. it did not spit out any error and I could set up the basic gateway configuration in the Phoscon App. However, everything related to the actual ConnBee hardware did not work. Phoscon showed the device version but not the firmware version and all ZigBee properties (channel etc.) were zeroed. I could not bind any sensors.

So I ended up flashing a left-over Raspberry Pi 3 with the official stable image. With that, everything now works fine and I got all my sensors integrated in a nice Grafana dashboard.

(OT: The Pi image has its own glitches, e.g. in its default configuration it tried to start hostapd in an endless loop which produced around 700 MiB of logs within a single week (poor SD card!) and a ton of unnecessary load. I'd be happy to send pull requests for improving the setup (I happen to make images for embedded devices as my day-time job), but haven't found a suitable repo.)

[MattL0]

One long term goal is to expose the GUI node parts via REST-API so that even for headless installations a remote client can display interactive nodes, with all features of deCONZ GUI, for example in a browser.

That would be a dream coming true

[nonplusnl]

Running into the exact same issue, trying to install deconz headless on Ubuntu server 22.04 (headless), but not happy to install 100+ of dependencies (>300MB) just for a GUI that I don't use. Is there any progress to report on this two-year-old request? Or perhaps some workaround to avoid at least the X11/Wayland dependencies? Many thanks!

[manup]

I'm not happy with these dependencies either, but they will be around for a while as everything happen in phases :/

Notes on the progress:

1. We started to trim down the REST-API plugin to remove device specific code from C++ side in favor to the handling via DDF. Ideally when finished there will be only a rather smallish C++ part left compared to the current code base.

Trimming the code base is quite a task and needed to be able to get rid of dependencies. Here are the current stats for lines of code in deCONZ core + plugins:

---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C++                                203          20593          10318         117084
C/C++ Header                       177           2823           3436          21420
Qt                                  27              1              0           8082
C                                    6            168            288           1097

2. Inside deCONZ core work has continued to decouple GUI from the logic, and slowly removing Qt usage for the logic handling for own dependency free implementation.

At some point when these two major tasks progressed enough the GUI, Qt UI libraries and their X11 dependencies will be removed, leaving only the core logic and the plugin – but still depending on Qt core libraries, replacing these for dependency free code is another major step.

3. The GUI itself becomes a separate application, as announced in v2.15.0-beta changelog I'm working on it daily and can already tell it forecasts how we can get rid of Qt in deCONZ core and the REST-API plugin.

The goal for the dependencies:

Overall the following dependencies can be expected to be installed on almost every system. For Qt beside the huge package size another draw back is that we have dozens of versions of Qt to deal with in the wild.

deCONZ / REST-API plugin (and upcoming Matter code):

• OpenSSL (or mbedtls)
• SQLite3

Capabilities that we currently rely on Qt libraries — like event, string, JSON, HTTP and WebSocket handling, will be replaced with dependency free code from the new GUI.

New GUI

• OpenGL and respective native OS windowing libraries
• OpenSSL (or mbedtls)
• Harfbuzz (optional)
• Freetype (optional)

GCFFlasher 4

see beta repository

• libgpiod (optional)

[lynix]

Running into the exact same issue, trying to install deconz headless on Ubuntu server 22.04 (headless), but not happy to install 100+ of dependencies (>300MB) just for a GUI that I don't use. Is there any progress to report on this two-year-old request? Or perhaps some workaround to avoid at least the X11/Wayland dependencies? Many thanks!

My workaround to this issue in the end was to setup Home Assistant in a Docker container. That one seems to be able to talk to my ConnBee directly, without going through deconz.

[PMunch]

Just want to chime in and say that on my system deconz pulled 750Mb of dependencies, again for a UI I won't ever see. Really quite annoying.