Enroll Linux on arm64 hosts to Fleet

[mikermcneil]

Goal

User story
As an endpoint operator,
I want to generate a fleetd agent w/ and w/o Fleet Desktop that I can install on Linux arm64
so that I can enroll my Linux hosts that run on arm64.

Product

• [ ] fleetctl changes: Universal fleetctl binary for Linux (this might already exist)
• [ ] fleetd changes: Publish all components of fleetd (osquery, Orbit, and Fleet Desktop) for Linux arm64 to the Fleet managed TUF server
• [x] Outdated documentation changes:
  • ~~Double check contributor docs and add mention of publishing fleetd for Linux arm64 to the TUF server~~
  • ~~Double check public docs and remove any mention of not being able to enroll Linux on arm64 hosts to Fleet~~

Engineering

• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

Context

• Requestor(s): _________________________

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.

[zmackie]

Little bit of clarification: we've started to investigate running on ARM architecture and its becoming a requirement for considering any solution. Osquery upstream does support ARM at this time.

[noahtalerman]

@zmackie is this a request for releasing Orbit for Linux ARM specifically?

Thank you for the above clarification.

[mikermcneil]

See also https://github.com/fleetdm/fleet/issues/1031

[zmackie]

One of our requirements for adopting fleet is the ability to deploy osquery to aarch64 hosts (amazon graviton). So, for us, the packaging commands will need to be able to produce something that targets that architecture. ie go run ./cmd/package/package.go --type deb --arch ARM64

As I understand it, upstream osquery binaries for that architecture are available, so my guess is that this would be a matter of

1. using those upstream binaries
2. compiling orbit to that architecture (easy in go - the compiler can do this)

[lizthegrey]

You may want to be able to manage M1 Mac ARM64 too, but +1 to do Linux ARM64 first.

[n8felton]

+1 for arm64 (M1/Apple Silicon) support on macOS. Preference would be a Universal 2 binary.

Goals:

• fleetctl package --type=pkg results in a installer package for macOS that deploys a Universal 2 binary of orbit and osqueryd

[lucasmrod]

We've created #4420 for the M1/Apple support. And this issue will be dedicated to Linux arm64 support.

[sharvilshah]

High level notes on how to go about building fleetctl, orbit and fleet-desktop on Linux aarch64:

• Figure out if Github Actions has an arm64 hosted runner available
  • If not, we will need to explore the self-host runner option
• Update MySQL image in docker-compose to support arm64
• Update orbit's goreleaser.yml
• Generate osqueryd and fleet-desktop targets in our workflows (generate-osqueryd-targets.yml and generate-desktop-targets.yml)
  • Sign and push these to TUF
• Update and add a new target in our Makefile along with the Dockerfile-desktop-linux
• Update fleetctl/package and orbit/pkg to add --arch flag or something similar

There might be some unknown unknowns here, this still needs a bit of digging deeper and prioritization

[zhumo]

Hello, given our current capacity, we will de-prioritize this in favor of other work. We will reconsider later in the quarter once we've met our other commitments or if this becomes a customer commitment.

[rhuddleston]

Here is a Makefile and Dockerfile that will build both amd64 and arm64 and verified it works:

Makefile:

VERSION=4.22.1
IMAGE=fleetdm/fleet:${VERSION}

.DEFAULT: buildx

buildx:
	@export DOCKER_CLI_EXPERIMENTAL=enabled
	@docker buildx build --build-arg VERSION=${VERSION} --platform linux/amd64,linux/arm64 -t ${IMAGE} --push .
	@echo
	@echo "The image has been built: ${IMAGE}"
	@echo

Dockerfile:

FROM --platform=$BUILDPLATFORM node:16 as website

ENV BUILD_ORG fleetdm
ENV PROJECT_NAME fleet
ARG VERSION

WORKDIR /opt/fleet
RUN git clone --depth 1 --branch fleet-v$VERSION https://github.com/$BUILD_ORG/$PROJECT_NAME.git /opt/$PROJECT_NAME && \
    make deps-js && make generate-js

FROM golang:1.19.3-alpine3.16 as builder

ENV GO111MODULE on

COPY --from=website /opt/fleet/ /opt/fleet
RUN apk update && \
    apk add ca-certificates git bash gcc musl-dev make && \
    git config --global http.https://gopkg.in.followRedirects true && \
    cd /opt/fleet && \
    make deps-go && make generate-go && make

FROM alpine:3.16

COPY --from=builder /opt/fleet/build/fleet* /usr/bin/
RUN apk --no-cache add ca-certificates && \
    ln -s /tmp /.goquery

CMD ["fleet", "serve"]

Glad to add it to the repo if there is a directory where it makes sense (maybe down tools)?

[zhumo]

hey @rhuddleston That's awesome. We'd like to start off by including that in our documentation. Someone from fleet (@chiiph?) will reach out about that.

I also created an issue that tracks adding your contribution to our dockerhub. https://github.com/fleetdm/fleet/issues/8904

[michalnicp]

@rhuddleston I tested your Makefile and Dockerfile and it looks pretty good. Unfortunately, I think we would need to update our ci workflows and make some significant changes to our existing Makefile and Dockerfile so that the multi-platform image gets pushed to dockerhub. All of our existing deployments user dockerhub as the docker registry. We will do this in #8904.

[RachelElysia]

Glad to add it to the repo if there is a directory where it makes sense (maybe down tools)?

@rhuddleston we got the green light to add your solution to the Deploying section of our docs. Since you offered and it's your solution, would you like to open a PR to contribute where you see fit in Deploying? We want to include a caveat that we don't support deploying using arm officially, but this is the suggested way in the meantime. :)

Feel free to add me, @chiiph, and @chris-mcgillicuddy as reviewers, or let me know if you just want me to add it to the documentation instead.

[rhuddleston]

Can we just have the documnetation link it to a directory in the fleet source with these two files?

[mike-j-thomas]

Hey @zhumo, please can you give me a little more info about what you need from #g-website team for this? I'm not sure if you are asking for us to create a directory for files in the comment above

Glad to add it to the repo if there is a directory where it makes sense (maybe down tools)?

Or if you only want us to add a reference in the docs?

We'd like to start off by including that in our documentation.

[mike-j-thomas]

Hey @zhumo, just a little nudge about what you need for this ⬆️

[zhumo]

Shoot, sorry @mike-j-thomas I think I mis-labeled this. Not sure what I was thinking.

[dherder]

@noahtalerman The ability to deploy Linux agents on arm64 is becoming critical for testing linux script execution (and the development of linux script libraries). Can we re-examine this, if just to build fleetd arm64 compatible package creation?

[alirezaghey]

Do I understanding this correctly, that there is currently no way to add Linux arm64 hosts to fleet?

[noahtalerman]

@alirezaghey yes. We're working on adding support for Linux arm64.

[noahtalerman]

@mikermcneil heads up, I moved your original issue description here:

As a user of osquery via Orbit (aka Fleet-osquery), I want a release of it available for Linux arm64.

How?

• Add --arch option (or introduce new --types, rpm-arm64 and deb-arm64) to allow fleetctl package to generate deb/rpm for the arm64 architecture.
• Osquery already provides support for Linux arm64. We would need to compile Orbit with such GOARCH.
• This also includes supporting upgrades of orbit and osqueryd for such architectures via Fleet's TUF system.
• We will also need to support compiling Fleet Desktop for linux/arm64.

[noahtalerman]

Hey @georgekarrv IIRC we spoke briefly about how we might accomplish this and you had a pretty good grasp of what we need to do.

If that's true, when you get the chance, can you please update the issue description w/ the required changes to enroll Linux on arm64 hosts?

[dherder]

more interest in arm64 support: https://macadmins.slack.com/archives/C0214NELAE7/p1708535372632249

[noahtalerman]

@dherder heads up, this user story is in the current design sprint (drafting) so we're targeting shipping this in the next 6 weeks. No need to bring it back to feature fest yet.

We'll update you if the story doesn't make the 3 week drafting timeline. At that point, we can bring it back to feature fest.

[georgekarrv]

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @ghernandez345 @gillespi314 @jahzielv @mna @roperzh

[dantecatalfamo]

With regards to cross-compiling CGO, I would highly suggest taking a look at zig cc if this hasn't already been solved. I've used it for cross-compiling my own CGO projects, and it makes builds across multiple targets and architectures substantially easier. Uber already uses it for this purpose.

[nonpunctual]

Duplicates: https://github.com/fleetdm/fleet/issues/18532

[noahtalerman]

Hey @nonpunctual heads up, this story is in the current engineering sprint (has the :release label and 4.50-tentative milestone.

Removing this story from feature fest.

cc @georgekarrv

[nonpunctual]

@noahtalerman when this closes https://github.com/fleetdm/fleet/issues/18532 can also be closed as dupe. Thanks.

[lucasmrod]

fleetctl changes: Universal fleetctl binary for Linux (this might already exist)

~TIL about universal binaries for Linux. Couldn't we do the same for fleetd components? (That is, build orbit and fleet-desktop as fat linux executables so that we don't need more targets to publish on every release. If possible, this would keep releasing fleetd simpler)~

EDIT: It seems FatELF is not a thing (not going to be supported by Linux kernel). https://icculus.org/finger/icculus?date=2009-11-03&time=19-08-04.