pipecd icon indicating copy to clipboard operation
pipecd copied to clipboard
verified publisher icon pipe-cd

Quickstart: PipeCD control plane does not work on Arm architecture cluster

Open ponkio-o opened this issue 1 year ago • 7 comments

What happened: Quickstart doesn't work on Arm architecture kubernetes cluster.

related: https://github.com/pipe-cd/pipecd/issues/4116

What you expected to happen: Working pipecd correctly on kubernetes cluster using arm arch.

How to reproduce it: Run pipectl quickstart against a Kubernetes cluster configured with only Arm architecture worker nodes.

My k8s cluster configured with only arm64 nodes.

$ k version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

$ k get node -o go-template='{{range .items}}{{.metadata.name}}  {{index .metadata.labels "kubernetes.io/arch"}}{{"\n"}}{{end}}'
instance-20211220-2249  arm64
instance-20211220-2250  arm64
instance-20211222-2325  arm64
instance-20211222-2332  arm64

Run pipectl quickstart

$ pipectl quickstart --version v0.47.0
Installing the controlplane in quickstart mode...
Release "pipecd" does not exist. Installing it now.
NAME: pipecd
LAST DEPLOYED: Wed Apr 24 11:29:25 2024
NAMESPACE: pipecd
STATUS: deployed
REVISION: 1
TEST SUITE: None

Intalled the controlplane successfully!

Waiting for PipeCD control plane to be ready...
PipeCD control plane status: ContainerCreating 3 Running 1 Init:0/1 2
PipeCD control plane status: ContainerCreating 1 CrashLoopBackOff 2 Running 1 Init:0/1 2
PipeCD control plane status: ContainerCreating 1 CrashLoopBackOff 2 Running 1 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: Running 3 Init:0/1 2 Error 1
PipeCD control plane status: Running 3 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
Failed to expose PipeCD control plane service!!
github.com/pipe-cd/pipecd/pkg/app/pipectl/cmd/quickstart.(*command).run
        /home/runner/work/pipecd/pipecd/pkg/app/pipectl/cmd/quickstart/quickstart.go:121
github.com/pipe-cd/pipecd/pkg/cli.runWithContext
        /home/runner/work/pipecd/pipecd/pkg/cli/cmd.go:95
github.com/pipe-cd/pipecd/pkg/cli.WithContext.func1
        /home/runner/work/pipecd/pipecd/pkg/cli/cmd.go:51
github.com/spf13/cobra.(*Command).execute
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:950
github.com/spf13/cobra.(*Command).Execute
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/pipe-cd/pipecd/pkg/cli.(*App).Run
        /home/runner/work/pipecd/pipecd/pkg/cli/app.go:69
main.main
        /home/runner/work/pipecd/pipecd/cmd/pipectl/main.go:48
runtime.main
        /opt/hostedtoolcache/go/1.20.5/x64/src/runtime/proc.go:250
2024/04/24 11:41:28 failed while waiting for server to be ready

Some components do not start properly.

$ k get pod -n pipecd
NAME                              READY   STATUS     RESTARTS      AGE
pipecd-cache-56c7c65ddc-qngvf     1/1     Running    0             35s
pipecd-gateway-58589b55f9-8jj9z   0/1     Error      2 (28s ago)   35s
pipecd-minio-677999d5bb-v2727     0/1     Error      2 (28s ago)   35s
pipecd-mysql-6db6688bd4-gnd82     1/1     Running    0             35s
pipecd-ops-667d6f6b74-5vvcg       0/1     Init:0/1   0             35s
pipecd-server-9bcf9f7fd-kvwb4     0/1     Init:0/1   0             35s

This is due to the fact that the container image used doesn't support arm arch. The following is an example for envoyproxy/envoy-alpine, but other Pods are similar. https://hub.docker.com/r/envoyproxy/envoy-alpine/tags/?page=&page_size=&ordering=&name=v1.18.3 CleanShot 2024-04-24 at 11 33 54

$ k logs pipecd-gateway-58589b55f9-8jj9z
exec /usr/local/bin/envoy: exec format error

$ k get pod pipecd-gateway-58589b55f9-8jj9z -o json | jq .spec.containers[].image
"envoyproxy/envoy-alpine:v1.18.3"

Upon investigation, the following container images do not support arm.

  • pipecd-gateway
    • https://hub.docker.com/r/envoyproxy/envoy-alpine/tags/?page=&page_size=&ordering=&name=v1.18.3
  • minio/minio
    • https://hub.docker.com/r/minio/minio/tags?page=&page_size=&ordering=&name=RELEASE.2020-08-26T00-00-49Z
  • alpine (InitContainer)
    • https://hub.docker.com/_/alpine/tags?page=&page_size=&ordering=&name=3.14
  • pipe-cd/pipecd
    • https://github.com/pipe-cd/pipecd/pkgs/container/chart%2Fpipecd/201285673?tag=v0.47.0

The pipe-cd/pipecd container image is built for arm64, but the pipecd binary inside the container appears to be built for amd64.

$ docker inspect ghcr.io/pipe-cd/pipecd:v0.47.0 | jq .[].Architecture
"arm64"

$ docker run --entrypoint sh --rm -it ghcr.io/pipe-cd/pipecd:v0.47.0
/ # which pipecd
/usr/local/bin/pipecd

$ docker ps
CONTAINER ID    NAME             IMAGE             TAG        STATE        STATUS           PUBLISHED PORTS
------------    -------------    --------------    -------    ---------    -------------    ---------------
dbfa86dacef5    goofy_ganguly    pipe-cd/pipecd    v0.47.0    [RUNNING]    Up 12 seconds
$ docker cp dbfa86dacef5:/usr/local/bin/pipecd ./

$ file pipecd
pipecd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=WUGpoqiNmaaNRUBLNkNh/JdKvMBjyQMmzFXHw0YM5/9e34ewlbEHqbW1yVSxxv/ZC6HmlDZ9SWAo1MGHTTC, not stripped

Environment:

  • piped version: v0.47.0
  • control-plane version: v0.47.0
  • Others:

ponkio-o avatar Apr 24 '24 02:04 ponkio-o

Hi @ponkio-o, Thanks for reporting this 👍 Basically, this can be renamed as PipeCD control plane does not work on Arm architecture cluster.

Also, the idea of PipeCD architecture does not require users to install the control plane in the same cluster as the applications cluster; only the piped agent should be installed in the same cluster with the application. So I think this should be only a nice-to-help enhancement. WDYT?

Of course, I understand that in some cases, users may only have Arm architecture clusters to install pipecd, but please understand my point of recommending not forcing the pipecd control plane in the same cluster as the applications cluster.

ref: https://pipecd.dev/docs-v0.47.x/concepts/

khanhtc1202 avatar Apr 24 '24 03:04 khanhtc1202

@khanhtc1202 Thank you for your response!

Also, the idea of PipeCD architecture does not require users to install the control plane in the same cluster as the applications cluster; only the piped agent should be installed in the same cluster with the application. So I think this should be only a nice-to-help enhancement. WDYT?

Does this mean that PipeCD is not designed to run the control plane and applications on the same cluster? Also, I feel like the issue of not supporting the Arm architecture and the issue of where the control plane operates are separate problems.

ponkio-o avatar Apr 24 '24 08:04 ponkio-o

Does this mean PipeCD is not designed to run the control plane and applications on the same cluster?

I mean, the PipeCD architecture doesn't force users to install its Control Plane in the same cluster as the application cluster, which is a good point since some applications require special specs for their infra. You can install PipeCD control plane in the same cluster with your application, but keeping ControlPlane separated from your applications cluster is the key point for PipeCD to support multi-platform deployment.

Also, I feel like the issue of not supporting the Arm architecture and the issue of where the control plane operates are separate problems.

I agree with this, too. What I want to mention here is instead of treating this issue as a bug, we should tag it as an enhancement issue 👍 I mention this point here in case you're starting with PipeCD but only have an Arm cluster; you can prepare another cluster for running the PipeCD control plane while installing the Piped agent to your arm arch application running cluster for testing in the meantime. That was my point 😄

khanhtc1202 avatar Apr 24 '24 09:04 khanhtc1202

I agree with this, too. What I want to mention here is instead of treating this issue as a bug, we should tag it as an enhancement issue 👍

Thank you. If that's the case, I think it's okay for me to handle it as an enhancement as well 👍 However, when I tried it, it did not work correctly with the pipectl quickstart command, so it might be a good idea to include a note in the documentation until full support for ARM is achieved.

ponkio-o avatar Apr 24 '24 10:04 ponkio-o

@ponkio-o I updated the label for the issue from bug -> enhancement. Thank you 👍 Also, do you want to propose a fix for this issue? 😄

khanhtc1202 avatar Apr 25 '24 10:04 khanhtc1202

Thank you for change the label! I'm going to investigate this issue.

ponkio-o avatar Apr 26 '24 01:04 ponkio-o

@ponkio-o Thank you so much. assigned you 👍

ffjlabo avatar Apr 26 '24 06:04 ffjlabo