compiling lxd-agent for FreeBSD

[igalic]

Required information

• Distribution: Ubuntu
• Distribution version: focal
• The output of "lxc info" or if that fails:
  • Kernel version: 5.15.0
  • LXC version: 5.13
  • LXD version: 5.13
  • Storage backend in use: dir

Issue description

I'm trying to integrate FreeBSD into cloud-init's CI. A requirement is that FreeBSD runs under LXD — that works, more or less. But I'd also like to get lxd VM agent running under FreeBSD.

Steps to reproduce

1. on a 14.0-CURRENT FreeBSD amd64 system
2. pkg install go (which gives us: go version go1.20.3 freebsd/amd64)
3. CGO_ENABLED=0 go install -v -tags agent,netgo github.com/lxc/lxd/lxd-agent@latest

and get the following build failure:

github.com/lxc/lxd/lxd/ip
# github.com/lxc/lxd/lxd/ip
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:55:26: undefined: syscall.NLM_F_REQUEST
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:55:50: undefined: syscall.NLM_F_ACK
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:94:55: undefined: syscall.NetlinkRouteAttr
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:143:51: undefined: syscall.NetlinkRouteAttr
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:169:61: undefined: nl.RtAttr
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:175:13: undefined: nl.Genlmsg
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:177:15: undefined: nl.GENL_CTRL_VERSION
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:180:12: undefined: nl.NewNetlinkRequest
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:180:36: f.ID undefined (type *netlink.GenlFamily has no field or method ID)
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:197:63: undefined: nl.RtAttr
go/pkg/mod/github.com/lxc/[email protected]/lxd/ip/vdpa.go:180:36: too many errors
github.com/lxc/lxd/lxd/storage/filesystem
# github.com/lxc/lxd/lxd/storage/filesystem
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:176:32: undefined: unix.MS_SYNCHRONOUS
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:177:32: undefined: unix.MS_NOATIME
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:178:31: undefined: unix.MS_BIND
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:180:32: undefined: unix.MS_NODEV
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:181:32: undefined: unix.MS_NODIRATIME
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:182:31: undefined: unix.MS_DIRSYNC
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:183:32: undefined: unix.MS_NOEXEC
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:184:31: undefined: unix.MS_LAZYTIME
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:185:31: undefined: unix.MS_MANDLOCK
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:186:31: undefined: unix.MS_NOATIME
go/pkg/mod/github.com/lxc/[email protected]/lxd/storage/filesystem/fs.go:186:31: too many errors

[igalic]

n.b.: FreeBSD 14.0-CURRENT has extensive Netlink support. But, go being go, can't know that, because it's duplicating the libc

[stgraber]

A bunch of that code isn't really needed inside of the agent, so selected build tags should be able to improve things.

The main catch is going to be the terminal handling code which may or may not already have FreeBSD support and the vsock code which similarly may or may not have support for FreeBSD. If those two behave, then this may be pretty trivial.

[igalic]

ref: https://github.com/golang/go/issues/59865

[stgraber]

If the kernel flags in the kernel headers on FreeBSD are named differently from their Linux equivalent, I'd actually argue that a low level module like unix should not try to rename them to line up with Linux.

Instead we need a higher level module (possibly LXD) to use the correct ones based on platform with the correct build tags set on the files to have them be pulled based on OS.

[stgraber]

But I'm also not convinced that lxd-agent needs to import all that logic to begin with, so we may be able to avoid this whole thing entirely by being a bit more clever about imports and file tagging.

[daniloegea]

I don't want to create any expectations, but:

[danilo@yoga ~]$ lxc exec -T freebsd-head -- uname -sr
FreeBSD 15.0-CURRENT
[danilo@yoga ~]$ lxc exec -T freebsd-head -- sysctl dev.vtsock
dev.vtsock.0.guest_cid: 102580108
dev.vtsock.0.%parent: virtio_pci4
dev.vtsock.0.%pnpinfo: vendor=0x00001af4 device=0x1053 subvendor=0x1af4 device_type=0x00000013
dev.vtsock.0.%location: 
dev.vtsock.0.%driver: vtsock
dev.vtsock.0.%desc: VirtIO VSOCK Transport Adapter
dev.vtsock.%parent: 

:smiley:

I've been working on this on my free time for several months now. It involved writing the FreeBSD drivers for the virtio_socket device and vsock, fixing issues in the p9fs driver, patching the Golang x/sys module, patching the vsock-go module and of course the lxd-agent itself. Now, everything is very unstable and look more like a proof of concept than anything. I'll continue to slowly work on this and maybe one day get it done.

[tomponline]

@daniloegea amazing, thank you! :)

[tomponline]

@daniloegea if you find that lxd-agent is dependent on something not strictly necessary, it maybe an incorrect import we can address to reduce the number of dependencies.