Support ARM

[ekcasey]

GOAL: Support buildpack builds for ARM images

** WARNING** Not yet ready to implement, may require RFCs, spec changes and/or imgutil changes.

Lifecycle Behavior

analyzer

1. If the previous image is a multiarch image, select the child manifest that matches the runtime architecture of the build environment.
2. Warn and proceed without analyzing the previous image if there the image exists but has the wrong architecture.

Because analyzer may run in a lifecycle image rather than in the build environment the runtime architecture of the analyze phase may not match the build environment. We need an RFC to either:

1. Record the information in detector and pass it to analyzer.
2. Accept an -arch flag and fail if the previous image is a multiarch image and the -arch flag is not provided by the platform

exporter

1. If the run-image is a multiarch image, select the child manifest that matches the runtime architecture of the build environment.
2. Fail if there is no run-image with the correct architecture
3. If there was a previous image, validate that the previous image architecture and the run image architecture match.

Because exporter may run in a lifecycle image rather than in the build environment the runtime architecture of the analyze phase may not match the build environment. We need an RFC to either:

1. Record the information in a previous phase and pass it to analyzer.
2. Accept an -arch flag and fail if the previous image is a multiarch image and the -arch flag is not provided by the platform

rebaser

1. If the run-image is a multiarch image, select the child manifest that matches the app image architecture
2. If the run-image is not a multiarch image rebaser should validate that the architecture matches the app-image before rebasing.

Lifecycle Artifacts

Publish ARM lifecycle artifacts

Add the following new Makefile targets:

• make build-linux-x86-64
• make build-linux-arm64
• make build-windows-x86-64
• make build-windows-arm64
• make package-linux-x86-64
• make package-linux-arm64
• make package-windows-x86-64
• make package-windows-arm64 Binaries should end up in out/<os>/<arch>.

And update the existing targets:

• make build-linux -> make build-linux-x86-64 && make build-linux-arm64
• make build-windows -> make build-windows-x86-64 && make build-windows-arm64
• make package-linux -> make package-linux-x86-64 && make package-linux-arm64
• make package-windows -> make package-windows-x86-64 && make package-windows-arm64

Local ARM image builds

Add the following new Makefile targets:

• make build-image-linux-amd64
• make build-image-linux-arm64
• make build-image should create a multiarch image containing a windows image and both linux architectures.

Docker desktop should support both building for multiple architectures. I do not know if a daemon running on a x86 linux machine will allow us to load arm64 images.

Note We cannot publish arm windows images becausemcr.microsoft.com/windows/servercore:ltsc2019 does not include an arm manifest (unlikemcr.microsoft.com/windows/nanoserver:1809 which we recently moved away from).

Publish ARM images

Instead of publish buildpacksio/lifecycle:<commit>-linux images for each build, publish

• buildpacksio/lifecycle:<commit>-linux-amd64
• buildpacksio/lifecycle:<commit>-linux-arm64

The buildpacksio/lifecycle:<commit> multiarch image should contain the windows image and both linux architectures.

When we release a new lifecycle version we should retag appropriate images with

• buildpacksio/lifecycle:<version>-linux-amd64
• buildpacksio/lifecycle:<version>-linux-arm64
• buildpacksio/lifecycle:latest-linux-amd64
• buildpacksio/lifecycle:latest-linux-arm64

And ensure that both buildpacksio/lifecycle:latest and buildpacksio/lifecycle:<version> include both linux architectures.

[dfreilich]

This issue focuses on support for arm64. There's an issue open by pack (https://github.com/buildpacks/pack/issues/907) re: arm64 and armv7 support – should I open a separate issue for armv7, or can we tack that on to this?

[dfreilich]

Also, for clarity sake, what RFC(s) would you imagine we would need to drive this through?

[natalieparellano]

Possibly relevant exploration around building ARM images on x86: https://github.com/buildpacks/samples/pull/60/files

[dfreilich]

Would we need separate lifecycle images for darwin/arm64 as well? If so, should I open up a new issue for that? Related: https://github.com/buildpacks/pack/issues/1003

[tibeer]

Ran into this issue today. Just for your information: I do not use lifecycle directly but rather by trying to use Hashicorps waypoint to spin up a Kubernetes deployment (of any container) on my local Kubernetes cluster. I am working on a M1 Mac.

[ryuheechul]

👍🏼 for this issue and I'm pretty much repeating what @tibeer said above.

Just to report a use-case, currently https://learn.hashicorp.com/tutorials/waypoint/get-started-docker?in=waypoint/get-started-docker would not work on M1 laptops with this error.

I can see there was this issue, https://github.com/buildpacks/pack/issues/1003 (in another repo) relating to m1 was resolved but that seems to be for the native binary building not container image building. And this, https://github.com/buildpacks/pack/issues/1171 also to seems be a related issue.

❯ waypoint up

» Building...
Creating new buildpack-based image using builder: heroku/buildpacks:18
✓ Creating pack client
❌ Building image
 │ 2021/06/10 21:32:23.990480 DEBUG:  Pulling image index.docker.io/heroku/buildpac
 │ ks:18

...

 │ 2021/06/10 21:32:27.168139 DEBUG:  -> heroku/[email protected]
 │ 2021/06/10 21:32:27.168142 DEBUG:  -> heroku/[email protected]
 │ 2021/06/10 21:32:27.168145 DEBUG:  -> heroku/[email protected]
 │ 2021/06/10 21:32:27.168150 DEBUG:  -> heroku/[email protected]
 │ 2021/06/10 21:32:27.168154 DEBUG:  -> evergreen/[email protected]
 │ 2021/06/10 21:32:28.220344 DEBUG:  Pulling image buildpacksio/lifecycle:0.11.3
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

[imjasonh]

I looked into this a bit out of curiosity, and it looked like building the lifecycle binary into an image would be fairly straightforward to accomplish with ko, not just for linux/arm64 but also more exotic architectures like linux/ppc64le -- basically anything go build supports, plus what you can find a base image for -- and after https://github.com/google/ko/pull/374 for Windows as well.

Essentially, ko does what your image build process already does: it go builds the binary for the desired output os+architecture, tars it up, appends it to a base image, and either puts it in the local daemon or pushes it to a registry.

The only hitch seems to be the use of cgo in priv/user_linux.go. If there's any way to avoid that, I think you could get multi-arch lifecycle images for lifecycle relatively easily with ko.

buildpacksio/pack doesn't have this issue, so you could get a multi-arch image for that even easier.

I'd be happy to help push this forward if this seems like a reasonable path. I understand the use of cgo might be unavoidable, or that changing the release process might be undesirable.

[micahyoung]

I feel like it's a great suggestion @imjasonh. Having more standardization and tooling around generating these foundational images would be a big win (selfishly, particularly for the Windows images). I'm not sure how maintainers might feel about migrating to ko, but if you're up for it, I think starting a separate issue here would be enough to kick off the discussion. I'll gladly add my +1.

[natalieparellano]

@ryuheechul it looks like your issue most closely relates to https://github.com/buildpacks/pack/issues/1197. Just to confirm, are you trying to build a linux/amd64 image? If so, I think a change in pack will unblock you. Is there any way to configure Waypoint to trust the provided builder?

There seem to be several threads on this issue, I wonder if it's worth breaking them out:

• Potential to compile lifecycle for linux/arm64 (maybe with ko?) in order to build linux/amd64 images
• Ability for lifecycle to build linux/arm64 images
• When using untrusted builders on an M1 Mac, ability for pack to pull the linux/amd64 lifecycle image (https://github.com/buildpacks/pack/issues/1197)

[natalieparellano]

Just wanted to let anyone interested know that there is an #arm channel in buildpacks Slack. This thread provides a somewhat mind-boggling matrix of possible supported scenarios. Please feel free to add your thoughts here!

[ryuheechul]

@natalieparellano Thanks for your comment. Btw, I'm not trying to build anything, all the logs are triggered by Waypoint itself and based on the error logs at the bottom,

 │ 2021/06/10 21:32:28.220344 DEBUG:  Pulling image buildpacksio/lifecycle:0.11.3
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

I think it's simply because the docker image is not built for linux/armX as I can see it here, https://hub.docker.com/r/buildpacksio/lifecycle/tags.

But then, I'm only guessing at this point so I might be wrong.

[warlockdn]

Do you guys have anything planned for arm architecture ? I also came across this issue while using waypoint.

[imjasonh]

After #659 (and #660 😓), the latest lifecycle image (buildpacksio/lifecycle:caab214) built from head now includes a linux/arm64 image:

$ crane manifest buildpacksio/lifecycle:caab214 | jq '.manifests[].platform'
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm64",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.17763.2029"
}

[warlockdn]

does this solve no matching manifest for linux/arm64/v8 issue ?

[natalieparellano]

@warlockdn I believe that issue is more likely to be solved by the fix for https://github.com/buildpacks/pack/issues/1197, which will be available in pack 0.20.0.

[warlockdn]

Wow. That's just great to hear. Looking forward. @natalieparellano 0.20 releases next ?

[natalieparellano]

@warlockdn yep! 0.20.0 should be out next week or the following.

[willianpaixao]

Hey @natalieparellano, pack-v0.20.0 was released but I couldn't find buildpacks/pack#1197 in the Changelog. Is that right?

[natalieparellano]

@willianpaixao I believe it's fixed by https://github.com/buildpacks/pack/pull/1216. Does upgrading resolve your issue? Please let us know!

[cihantas]

@natalieparellano It appears like the issue isn't resolved. Using v0.11.4 on an M1 Mac still results in the same error message.

[natalieparellano]

@cihantas just confirming you are using pack v0.20.0 (v0.11.4 is the lifecycle version).

Edit: also just to confirm - are you trying to build an amd64 image?

[cihantas]

@natalieparellano Yes, I can confirm I'm using v0.11.4.

I'm using pack in the context of Hashicorp Waypoint. Not sure if I'm building the amd64 version. Is there a way for me to verify this?

[aemengo]

@cihantas Thanks for calling this out. This is a great find.

It seems that there's been some confusion (myself included) about how Hashicorp Waypoint works. Upgrading the pack CLI on your local machine won't have any effect because Waypoint is using pack via its golang API library. As of right now, Waypoint's pack version is pinned to v0.18.1.

There's the quicker fix which is to make a PR bumping Waypoint's pinned pack version to v0.20.0, like @natalieparellano suggested.

But I think you've highlighted a nascent problem with the pack Golang API library: the "trusted builder code path" isn't exercised at all. For trusted builders (like their default heroku/buildpacks:18), the lifecycle shouldn't even be downloaded. And more importantly, it means that the lifecycle's newer creator workflow isn't getting exercised.

--

If no one else wants to have the fun, I'll make the appropriate PRs in Waypoint, and pack, GitHub repositories - within the next couple days.

---

EDIT:

• hashicorp/waypoint#2023
• buildpack/pack#1254

[aemengo]

@cihantas @natalieparellano and other interested folks,

Just wanted to highlight that hashicorp released waypoint v0.5.0+ which should have the architecture fix. The following error should no longer be present:

 # ...
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

[imjasonh]

buildpacksio/lifecycle is still not released with arm64 support: https://explore.ggcr.dev/?image=buildpacksio%2Flifecycle

#680 should fix this (after another release)

[donnyv12]

another M1 Mac user who would love to use buildpacks here :wave:

[imjasonh]

As of v0.12.0-rc.1, the lifecycle image includes a linux/arm64 build -- the image is buildpacksio/lifecycle:0.12.0-rc.1.

For Waypoint users on M1, I'm not sure how to configure Waypoint to use this image, or if it's even configurable. If someone can manage to figure that out and try out the 0.12 RC and report back, that would be really helpful.

edit: Or will this require pack to bump its dep on lifecycle, and waypoint to bump its dep on pack, and do a release? 🤷

[aemengo]

@imjasonh The Waypoint issue is orthogonal to this "Support ARM" issue (and should be resolved with the latest Waypoint release). Let's please open another issue about Waypoint if someone is still finding issue.

[agracey]

What's the status of this? I see a lot of arm related build steps and artifacts in the makefile. If I build a builder with the arm binaries and ran it on a cluster that's arm only, would it fail?

[natalieparellano]

@agracey thanks for the feedback - we should probably close this issue or update it to include only the work yet to be done. If you have a builder where the base image, lifecycle, and buildpacks are all based on arm, and you run it on arm, it should produce an arm image.

The work still to be done is of the nature of edge cases that may arise when e.g., the lifecycle is running on one architecture but the desired output image is of another architecture. There is a matrix of possibilities that we need to evaluate to determine if they can be supported.