Odd `incus exec` regression with VMs in Debian packaging

[gibmat]

Is there an existing issue for this?

• [x] There is no existing issue for this bug

Is this happening on an up to date version of Incus?

• [x] This is happening on a supported version of Incus

Incus system details

Seen in both Incus 6.0.5 and 6.17.

Reported upstream as bug [#1114918](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1114918).

Instance details

No response

Instance log

No response

Current behavior

No response

Expected behavior

No response

Steps to reproduce

Commit 4e828cafc1469eeb1b257884f49b429aba9c5565 seems to have introduced some sort of regression that breaks running incus exec against a VM. When doing so the message "Error: websocket: bad handshake" is returned. If trying with a container, it works just fine.

Making this more interesting, if the incus-agent binary that's running within the VM is compiled "normally" via

CGO_LDFLAGS_ALLOW="-Wl,-z,now" CGO_ENABLED=0 go build -trimpath -tags=agent,netgo ./cmd/incus-agent/

it works, but when compiled using the dependencies as packaged by Debian it breaks

CGO_LDFLAGS_ALLOW="-Wl,-z,now" CGO_ENABLED=0 GOPATH=/usr/share/gocode/:/home/gibmat/ GOTOOLCHAIN=local GO111MODULE=off go build -trimpath -tags=agent,netgo ./cmd/incus-agent/

The first obvious check was if any libraries that Incus depends on are terribly out of date in Debian, and nothing really jumped out at me. The commit I bisected down to actually removes the use of gorilla/mux in favor of built-in libraries.

I'm not sure what else to try poking at to figure out why this is breaking for Debian's packaging. If it was a "real" regression (for lack of a better word), I'd expect the Zabbly packaging to also be experiencing this issue.

[stgraber]

I've seen a similar report on another distro where the same behavior would happen with a dynamically linked incus-agent.

Can you confirm that in your case the resulting binary is correctly static? The flags you're passing certainly suggest a static binary should be produced...

[gibmat]

Yes, the incus-agent binaries produced are statically linked. Also, both the host and VM are running up-to-date sid.

[alandoherty]

I'm also getting this error using incus exec with VM's running Fedora 42 Cloud, Ubuntu Noble Desktop and Ubuntu Plucky Desktop. Binaries seem to be dynamically linked. Host is Fedora 42.

Had to move to LXD. Let me know if I can provide more details.

[stgraber]

If you look at our releases, they do include static builds of the agent: https://github.com/lxc/incus/releases/tag/v6.17.0

Replacing the incus-agent on your system with the static one from the release, then restarting your VM should do the trick until you your distro's packager can fix their build.

[PACordonnier]

Hey I've investigate the issue and I'm quite sure its not related to dynamically or statically linked incus-agent binary.

I've tried incus-agent 6.15 static and dynamic running on a ubuntu 24.04 guest and same symptoms as this issue. Bug is not present using incus-agent 6.14 nor 6.16. So it has been fixed somehow but couldnt pinpoint where exactly.

For fedora 42 users such as me I think the solution will come when package will be updated or change the incus-agent binary somehow with a more recent one

but you're right @stgraber, fedora packages the rpm and builds the agent dynamically (as of today). Its incorrect but I dont think the issue is related to this

edit: bug is also fixed on 6.17 (static, binary found in this repo releases)

[alandoherty]

Can confirm I was running Incus 6.15, I have switched to LXD for now but can give it another try when 6.16 is available.

[stgraber]

@gibmat do you still have the issue if building the current Incus? If so, can you share some more reproduction steps so I can try to reproduce the environment here in a Debian VM?

[gibmat]

Yes, I'm still seeing this with the 6.18 release (not yet uploaded to experimental in Debian, since I'm waiting on a few new dependencies needed for building the MacOS/Windows agents to make it through the NEW queue).

Here's steps to easily reproduce the issue:

Start with a clean Debian sid VM and install incus (sudo apt install incus). This will install the LTS packaging, which is fine, since we'll reproduce the issue with incus-agent from the current main branch of the Incus repo.

Create an Incus VM and verify we can exec into the running instance. (The Debian packaging currently reverts commit 4e828cafc1469eeb1b257884f49b429aba9c5565.)

$ incus launch images:debian/trixie --vm debian
$ incus exec debian bash
root@debian:~#

Now, back on the main VM install the necessary build dependencies:

$ sudo apt install git
$ sudo apt build-dep incus

Create some expected directories and clone the Incus repo:

$ mkdir -p ~/src/github.com/lxc/
$ cd ~/src/github.com/lxc/
$ git clone https://github.com/lxc/incus.git
$ cd incus/

Building incus-agent like is done during the Debian package build process DOESN'T work:

$ CGO_LDFLAGS_ALLOW="-Wl,-z,now" CGO_ENABLED=0 GOPATH=/usr/share/gocode/:~ GOTOOLCHAIN=local GO111MODULE=off go build -trimpath -tags=agent,netgo ./cmd/incus-agent/
$ sudo mv incus-agent /usr/libexec/incus/incus-agent.linux.x86_64
$ incus restart debian
$ incus exec debian bash
Error: websocket: bad handshake

Building incus-agent "normally" DOES work:

$ CGO_LDFLAGS_ALLOW="-Wl,-z,now" CGO_ENABLED=0 go build -trimpath -tags=agent,netgo ./cmd/incus-agent/
$ sudo mv incus-agent /usr/libexec/incus/incus-agent.linux.x86_64
$ incus restart debian
$ incus exec debian bash
root@debian:~#

[breml]

I have the same issue, similar to what @alandoherty describes (Bluefin Linux, 42.20251021.1 (Silverblue)). For me also VM's do not work. In particular I have tested Debian 13 and Ubuntu 24.04. Bluefin currently ships with Incus 6.15.

[stgraber]

@breml Bluefin should have the issue that its incus-agent is dynamically linked. You can check with file. If it is, then fetch the latest incus-agent from the Github release and replace it with that and restart Incus and the VM, that should fix it.