skaffold icon indicating copy to clipboard operation
skaffold copied to clipboard

feat: cross-platform build support in GoogleCloudBuild

Open gsquared94 opened this issue 3 years ago • 4 comments

Fixes: #7555 Description

This PR implements support for running cross-platform builds on GoogleCloudBuild. It does this by injecting a prebuild step platformEmulatorInstallStep that runs docker/binfmt image to install required tooling for QEMU emulation on the GoogleCloudBuild containers. The user can also provide any alternate image explicitly in the googleCloudBuild.platformEmulatorInstallStep of the skaffold.yaml config.

Testing instructions:

  • Use the project go-guestbook.
  • Set the active kubernetes context to an ARM node cluster (googlers can use this cluster).
  • Run:
skaffold dev --default-repo=gcr.io/k8s-skaffold -p cloudbuild

Expected: Skaffold should build linux/arm64 images on GoogleCloudBuild and deploy to the active k8s cluster.


Follow-up Work (remove if N/A)

Add documentation and sample project.

gsquared94 avatar Aug 04 '22 09:08 gsquared94

Codecov Report

Merging #7739 (98f6c46) into main (290280e) will decrease coverage by 3.78%. The diff coverage is 53.59%.

@@            Coverage Diff             @@
##             main    #7739      +/-   ##
==========================================
- Coverage   70.48%   66.70%   -3.79%     
==========================================
  Files         515      587      +72     
  Lines       23150    28267    +5117     
==========================================
+ Hits        16317    18855    +2538     
- Misses       5776     8039    +2263     
- Partials     1057     1373     +316     
Impacted Files Coverage Δ
cmd/skaffold/app/cmd/credits/export.go 0.00% <0.00%> (ø)
cmd/skaffold/app/cmd/deploy.go 40.90% <0.00%> (-12.94%) :arrow_down:
cmd/skaffold/app/cmd/test.go 44.44% <0.00%> (ø)
cmd/skaffold/skaffold.go 0.00% <0.00%> (ø)
cmd/skaffold/app/cmd/inspect_tests.go 62.50% <14.28%> (-1.14%) :arrow_down:
cmd/skaffold/app/cmd/render.go 35.48% <18.18%> (-5.90%) :arrow_down:
cmd/skaffold/app/cmd/lsp.go 28.12% <28.12%> (ø)
cmd/skaffold/app/cmd/run.go 64.28% <33.33%> (-9.63%) :arrow_down:
cmd/skaffold/app/cmd/fix.go 56.41% <37.50%> (-20.07%) :arrow_down:
cmd/skaffold/app/cmd/verify.go 41.17% <41.17%> (ø)
... and 347 more

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

codecov[bot] avatar Aug 04 '22 10:08 codecov[bot]

Pending manual verification.

tejal29 avatar Aug 04 '22 19:08 tejal29

In attempting the manual testing steps above I am seeing:

Build [go-guestbook-frontend] failed: creating bucket if not exists: getting bucket "k8s-skaffold_cloudbuild": Get "https://storage.googleapis.com/storage/v1/b/k8s-skaffold_cloudbuild?alt=json&prettyPrint=false&projection=full": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform" not defined

Not sure if this is related to the added code or some IAM rules/perms I need but don't have related to the k8s-skaffold project.

EDIT: I believe this is related to the SA that I am using to kickoff the Cloud Build job not having sufficient perms to do a specific action in Cloud Build (although not 💯 sure). I am not using any SA directly though, just my aprindle account and auth via gcloud auth login

I am able to run & deploy with cloudbuild using amd64 with the current code when pointing to an amd64 cluster (in project-id:aprindle-test-cluster) and using aprindle-test-cluster cloud-build/default-repo

aaron-prindle avatar Aug 09 '22 18:08 aaron-prindle

Talked offline w/ @renzodavid9 who was able to get this working with the steps above. I believe this is some error with my gcloud or pantheon configuration.

aaron-prindle avatar Aug 09 '22 21:08 aaron-prindle