Consider publishing arm 64 `pulumi-kubernetes-operator` images

[phillipedwards]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Consider publishing arm 64 pulumi-kubernetes-operator images

Affected area/feature

[rquitales]

Note: we'll also need to verify that these images run as expected on ARM nodes.

[rquitales]

Upon further testing, this requires upstream changes to pulumi/pulumi-docker-containers to ensure that certain binaries work as expected.

I've pushed an alpha build of this image that could be used for verification and testing, and can be used with the following image tag: pulumi/pulumi-kubernetes-operator:v1.11.5-arm64-alpha.

[devantler]

Any news? 👀 Its been a year since this issue last saw activity.

[EronWright]

Update: in 2.x, we now do publish an arm64 build of the operator image (link). However, one still needs to have an arm64 build of the "kitchen-sink" pulumi image (see https://github.com/pulumi/pulumi-docker-containers/issues/297). While we do have arm64 builds of the language-specific images, we don't use them by default.

Also, we don't have "non-root" variants of the language-specific images (see https://github.com/pulumi/pulumi-docker-containers/issues/308).

[project0]

I just faced this problem within a ARM cluster:

Failed to pull image "pulumi/pulumi:latest-nonroot": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/pulumi/pulumi:latest-nonroot": no match for platform in manifest: not found

[project0]

I think for most use cases (with the program CRD) adding a image with the yaml-language provider (multi-arch) would be sufficient to enable support. Would it make sense to add a new image to https://github.com/pulumi/pulumi-docker-containers

For everyone else who just wants to get it running with arm64, here is a workaround with workspace template to get the yaml language provider (and therefore the native Program) running:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: example-stack
spec:
  serviceAccountName: pulumi-workspace
  config: {}
  envRefs: {}
  destroyOnFinalize: true
  programRef:
    name: program
  stack: "my-stack"
  workspaceTemplate:
    spec:
      image: pulumi/pulumi-go:latest
      securityProfile: baseline
      podTemplate:
        spec:
          volumes:
            - name: custom-providers
              emptyDir: {}
          initContainers:
            - name: install-yaml
              image: alpine:latest
              command: ["/bin/sh", "-c"]
              args:
                - arch=$(uname -m);
                  if [ "$arch" = "x86_64" ]; then arch="amd64"; fi;
                  if [ "$arch" = "aarch64" ]; then arch="arm64"; fi;
                  echo "Installing YAML Provider... ${arch}";
                  wget https://github.com/pulumi/pulumi-yaml/releases/download/v1.13.0/pulumi-language-yaml-v1.13.0-linux-${arch}.tar.gz -O - | tar -xz -C /custom-providers -f -;
                  echo "Done.";
              volumeMounts:
                - mountPath: /custom-providers
                  name: custom-providers
          containers:
            - name: pulumi
              volumeMounts:
                - mountPath: /pulumi/bin/pulumi-language-yaml
                  name: custom-providers
                  subPath: pulumi-language-yaml

[EronWright]

Thanks @project0 for the clever workaround.

[EronWright]

This ticket is still open to solve the "mismatch" problem that is mentioned above.

Failed to pull image "pulumi/pulumi:latest-nonroot": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/pulumi/pulumi:latest-nonroot": no match for platform in manifest: not found

There's at least three solutions:

1. Don't publish an arm64 build of pulumi-kubernetes-operator image.
2. Use a node affinity rule when a non-multi-arch image is being used. Maybe introduce an easier way to do that.
3. Publish multi-arch builds for all pulumi images.

When one is using an architecture-specific pulumi image, perhaps one should also use a node affinity rule on the workspace pod spec to target that architecture. Some pulumi images are multi-arch, some aren't, so it isn't clear how to automate this.

For example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
          - key: "kubernetes.io/arch"
            operator: In
            values: ["amd64"]

[EronWright]

Closing this ticket as complete, and tracking the outstanding "mismatch" issue in a separate ticket: https://github.com/pulumi/pulumi-kubernetes-operator/issues/818

[pulumi-bot]

Cannot close issue:

• does not have required labels: resolution/

Please fix these problems and try again.

[pulumi-bot]

Cannot close issue:

• does not have an assignee

Please fix these problems and try again.

[pulumi-bot]

Cannot close issue:

• does not have required labels: kind/

Please fix these problems and try again.