Kong
Building binary on linux/arm64 builder takes too long
Running go commands on linux/arm64 builder in release action takes a very long time, making the duration of build-push-images steps longered to ~20mins.
The go commands takes a very long time on linux/arm64 builder:
77s for go mod download (18s on linux/amd64 builder)
#16 [linux/arm64 builder 6/9] RUN go mod download
#16 DONE 77.0s
#17 [linux/amd64 builder 6/9] RUN go mod download
#17 DONE 18.6s
about 14 min to build binary (2 min on linux/amd64 builder)
20 [linux/amd64 builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH="amd64" GO111MODULE=on go build -a -o manager -ldflags "-s -w -X github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata.Release=2.4.0 -X github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata.Commit=e55100c041f2647c3e0b4db2bcbc1489950e808d -X github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata.Repo=https://github.com/Kong/kubernetes-ingress-controller.git" ./internal/cmd/main.go
#20 DONE 144.1s
#23 [linux/arm64 builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH="arm64" GO111MODULE=on go build -a -o manager -ldflags "-s -w -X github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata.Release=2.4.0 -X github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata.Commit=e55100c041f2647c3e0b4db2bcbc1489950e808d -X github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata.Repo=https://github.com/Kong/kubernetes-ingress-controller.git" ./internal/cmd/main.go
#23 DONE 835.3s
Is it caused by the performance difference between the builders? Is it possible to run cross compilation on linux/amd64 builder to build arm64 images?
Impact: makes the release process run for 20 minutes longer. Runs before the e2e tests (that are recently flaky).
Fixing the root cause (slow arm64 build) may be difficult due to the performance of arm64 builders available to us today; the low hanging fruit that makes the release process more interactive would be for the tests not to wait for the arm64 build to finish.
@randmonkey, would you please capture that ordering fix in a separate issue? thx
In nightly build jobs and release jobs, we build images before we run integration tests on the code base. So, the test part does not depend on the image built from actions. So we could move the image build step to either:
- be parallel with running tests (if GH action supports)
- be after integration tests passed The issue for changing the order: #2744
It seems to not be an issue anymore, otherwise, it'd be probably fixed already. Closing
