ErrImagePull for Public Apps on Kubernetes due to Invalid Image Name/Registry

[Kelvinloucosta]

Shuffle Version: 2.1.1

When running Shuffle on a Kubernetes cluster (specifically GKE, but likely affects all K8s installations), attempting to activate and use a public app, such as the official Okta app, consistently fails. The workflow execution fails because the Orborus worker tries to create a pod for the app, but this fails with an ErrImagePull status.

The root cause is that the system attempts to pull the image from an incorrect registry or with a non-existent tag. For example, for the Okta app, it tries to pull docker.io/frikky/shuffle:okta_1.1.0, which does not exist. The frikky user publishes images to ghcr.io, not docker.io, and the tag format seems incorrect.

Events: Type Reason Age From Message

---

Normal Pulling 2m48s (x5 over 5m54s) kubelet Pulling image "docker.io/frikky/shuffle:okta_1.1.0" Warning Failed 2m47s (x5 over 5m53s) kubelet Failed to pull image "docker.io/frikky/shuffle:okta_1.1.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/frikky/shuffle:okta_1.1.0": failed to resolve reference "docker.io/frikky/shuffle:okta_1.1.0": docker.io/frikky/shuffle:okta_1.1.0: not found Warning Failed 2m47s (x5 over 5m53s) kubelet Error: ErrImagePull

The on-premise download API (/api/v1/download_docker_image) is not a viable workaround for free-tier users, as it returns a 403 Forbidden error, indicating it's a feature for paid subscribers.

The source code for some public apps, like Okta, is not available in the public shuffle-apps repository, making it impossible to build the image manually.

This issue effectively prevents on-premise Kubernetes users from using several official, public apps.

Could the image path for public apps be corrected to point to the valid location (e.g., ghcr.io)? Alternatively, could the source code for these apps be made public in the shuffle-apps repository to allow users to build their own images? Thank you!

[frikky]

@0x0elliot

[alessio1983lx]

is there any workaround? I see in worker pod there are these envs:

SHUFFLE_BASE_IMAGE_TAG_SUFFIX=0.8.70 SHUFFLE_BASE_IMAGE_REGISTRY=docker.io SHUFFLE_BASE_IMAGE_NAME=frikky/shuffle

but i cannot understand where and if i can modify them in the helm values...

[yashsinghcodes]

Hey, follow this docs and it will solve the issue

https://shuffler.io/docs/configuration#private-apps-on-kubernetes

[Kelvinloucosta]

here are my configs:

apiVersion: apps/v1 kind: Deployment metadata: name: shuffle-backend namespace: shuffle spec: replicas: 1 selector: matchLabels: app: shuffle-backend strategy: type: Recreate template: metadata: labels: app: shuffle-backend spec: serviceAccountName: shuffle-backend-sa # <-- ✅ ESTA LINHA FOI ADICIONADA securityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 runAsNonRoot: true seccompProfile: type: RuntimeDefault initContainers: # Init Container: Espera o OpenSearch ficar pronto - name: wait-for-opensearch image: curlimages/curl:8.4.0 command: ['sh', '-c', 'until curl -fs http://shuffle-opensearch-service:9200/_cluster/health; do echo "Waiting for OpenSearch..."; sleep 5; done; echo "OpenSearch is ready!"'] securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1000 capabilities: drop: ["ALL"] seccompProfile: type: RuntimeDefault volumeMounts: - name: shuffle-apps mountPath: /shuffle-apps

  containers:
  - name: shuffle-backend
    image: us-central1-docker.pkg.dev/sauter-soc/sauter-soc-manager/shuffle-backend-custom:v3
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 5001
      name: http
    env:
    # --- ✅ CORREÇÕES APLICADAS ---
    - name: SHUFFLE_APP_BUILD_DISABLED
      value: "true"
    - name: SHUFFLE_HEALTHCHECK_DISABLED
      value: "true"

    # --- Configurações Básicas do Shuffle ---
    - name: SHUFFLE_OPENSEARCH_URL
      value: "http://shuffle-opensearch-service:9200"
    - name: SHUFFLE_OPENSEARCH_PASSWORD
      valueFrom:
        secretKeyRef:
          name: shuffle-secrets
          key: opensearch-password
    - name: SHUFFLE_DEFAULT_USERNAME
      value: "admin"
    - name: SHUFFLE_DEFAULT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: shuffle-secrets
          key: admin-password
    - name: SHUFFLE_DEFAULT_APIKEY
      valueFrom:
        secretKeyRef:
          name: shuffle-secrets
          key: shuffle-uuid
    - name: SHUFFLE_ENCRYPTION_MODIFIER
      valueFrom:
        secretKeyRef:
          name: shuffle-secrets
          key: shuffle-uuid
    - name: BASE_URL
      value: "http://shuffle-backend-service.shuffle.svc.cluster.local:5001"
    - name: SHUFFLE_BASE_URL
      value: "http://shuffle-backend-service.shuffle.svc.cluster.local:5001"
    - name: OUTER_HOSTNAME
      value: "https://shuffle.sautersec.com"
    - name: SHUFFLE_APP_HOTLOAD_LOCATION
      value: "/shuffle-apps"
    - name: SHUFFLE_APP_HOTLOAD_FOLDER
      value: "/shuffle-apps"
    - name: SHUFFLE_FILE_LOCATION
      value: "/shuffle-files"
    - name: SHUFFLE_INTERNAL_ONPREM
      value: "true"
    - name: ENVIRONMENT_NAME
      value: "Shuffle"
    - name: TZ
      value: "America/Sao_Paulo"
    - name: SHUFFLE_CLOUDRUN_CHECK
      value: "false"

    # --- Kaniko para Build de Apps ---
    - name: SHUFFLE_DOCKER_BUILD_TYPE
      value: "kaniko"
    - name: SHUFFLE_DOCKER_REGISTRY_URL
      value: "us-central1-docker.pkg.dev"
    - name: SHUFFLE_DOCKER_REGISTRY_PROJECT_ID
      value: "sauter-soc"
    - name: SHUFFLE_DOCKER_REGISTRY_REPOSITORY
      value: "sauter-soc-manager"
    - name: RUNNING_MODE
      value: "kubernetes"
    - name: IS_KUBERNETES
      value: "true"

    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "768Mi"
        cpu: "500m"
    volumeMounts:
    - name: shuffle-apps
      mountPath: /shuffle-apps
    - name: shuffle-files
      mountPath: /shuffle-files
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      runAsUser: 1000
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault
    startupProbe:
      httpGet:
        path: /api/v1/health
        port: 5001
      failureThreshold: 30
      periodSeconds: 10
    livenessProbe:
      httpGet:
        path: /api/v1/health
        port: 5001
      initialDelaySeconds: 10
      periodSeconds: 30
    readinessProbe:
      httpGet:
        path: /api/v1/health
        port: 5001
      initialDelaySeconds: 5
      periodSeconds: 15
  volumes:
  - name: shuffle-apps
    persistentVolumeClaim:
      claimName: shuffle-apps-pvc
  - name: shuffle-files
    emptyDir: {}

---

apiVersion: apps/v1 kind: Deployment metadata: name: shuffle-orborus namespace: shuffle spec: replicas: 1 selector: matchLabels: app: shuffle-orborus template: metadata: labels: app: shuffle-orborus spec: serviceAccountName: shuffle-orborus-sa securityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: orborus image: ghcr.io/shuffle/shuffle-orborus:2.1.1 imagePullPolicy: IfNotPresent env: - name: BASE_URL value: http://shuffle-backend-service.shuffle.svc.cluster.local:5001

    - name: SHUFFLE_NAMESPACE
      value: "shuffle"

    - name: SHUFFLE_ORBORUS_NAMESPACE
      value: "shuffle"

    - name: SHUFFLE_WORKER_URL
      value: "http://shuffle-workers.default.svc.cluster.local:33333"

    - name: SHUFFLE_ORBORUS_AUTH
      valueFrom:
        secretKeyRef:
          name: shuffle-secrets
          key: shuffle-uuid

    - name: RUNNING_MODE
      value: "kubernetes"

    - name: IS_KUBERNETES
      value: "true"

    - name: SHUFFLE_SKIP_PIPELINES
      value: "false"

    - name: SHUFFLE_WORKER_IMAGE
      value: "ghcr.io/shuffle/shuffle-worker:2.1.1"

    - name: SHUFFLE_WORKER_VERSION
      value: "2.1.1"

    - name: SHUFFLE_WORKER_SERVICEACCOUNT
      value: "shuffle-orborus-sa"

    - name: SHUFFLE_ORBORUS_EXECUTION_TIMEOUT
      value: "600"

    - name: SHUFFLE_ORBORUS_EXECUTION_CONCURRENCY
      value: "10"

    - name: ENVIRONMENT_NAME
      value: "Shuffle"

    - name: CLEANUP
      value: "true"

    - name: TZ
      value: "America/Sao_Paulo"

    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "256Mi"
        cpu: "200m"

    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      runAsUser: 1000
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault

[yashsinghcodes]

Hey @Kelvinloucosta, You need to set up a registry so that when the backend builds your app image, it has a place to push it and later pull it from. Follow this guide to set it up and you should be good to go.

The public images are only available for the apps here: https://hub.docker.com/r/frikky/shuffle All other app images are built on the fly by the backend, and in Kubernetes we need a Docker registry to support this.

[alessio1983lx]

Hey, follow this docs and it will solve the issue

https://shuffler.io/docs/configuration#private-apps-on-kubernetes

hi, i haven't used a private registry yet... anyway i see that when it tries to pull shuffle-tools this is the outcome in Kubernetes:

Normal BackOff 7s (x611 over 140m) kubelet Back-off pulling image "docker.io/frikky/shuffle/shuffle-tools:1.2.0"

i see that the image is not available anymore as docker.io/frikky/shuffle/shuffle-tools:1.2.0 but as docker.io/frikky/shuffle-tools:1.2.0

......

So, even setting up a local registry how can the system pull it for the first time? should i manually sync all the registry with scopeo and just use a private one?

The pull from within shuffle components is going to be fixed anyway?

Thanks for all the effort!

[valpet93]

The same here

[P4sca1]

@alessio1983lx @valpet93 Recent versions of shuffle have changed the image format from frikky/shuffle:app_version to frikky/shuffle/app:version. Ensure you are using the latest version of Shuffle worker.

[alessio1983lx]

@alessio1983lx @valpet93 Recent versions of shuffle have changed the image format from frikky/shuffle:app_version to frikky/shuffle/app:version. Ensure you are using the latest version of Shuffle worker.

hi P4sca1 ... are sure it is not the other way around? changed from frikky/shuffle/app:version to frikky/shuffle:app_version ... I see it ise trying frikky/shuffle/app:version .... But the repo does not exist, it is now just frikky/app (for example see shuffle-tools)...

I am using 2.1.0 because i see in the main page in github is the latest stable officially released (or maybe i am assuming)... Are you suggesting to use latest tag or nightly when installing via helm? Can you eventually point out a safe version for each component? (orborus, worker, frontend, backend)

Regards and thank you for the reply.

[P4sca1]

You are right, it is now frikky/app:version (not frikky/shuffle/app:version). Previous versions used frikky/shuffle:app_version. I'm currently running version 2.1.2 of every component.

[alessio1983lx]

You are right, it is now frikky/app:version (not frikky/shuffle/app:version). Previous versions used frikky/shuffle:app_version. I'm currently running version 2.1.2 of every component.

ok... i am testing... Anyway i had to leave tag 2.1.0 for the frontend... 2.1.2 seem existing just for the other components...

Anyway with 2.1.2 for everything (and 2.1.0 in frontend) i see in backend spamming this log message connecting to opensearch:

[WARNING] Failed to get current org : status: 405, error: Incorrect HTTP method for uri [/organizations/_doc/] and method [GET], allowed: [POST]

[alessio1983lx]

hi P4sca1 ... using 2.1.2 tries to pull images with the wrong name

frikky/shuffle/shuffle-tools:1.2.0

if i manually edit the pod removing /shuffle/ shuffle-tools is then deployed...

[P4sca1]

[WARNING] Failed to get current org : status: 405, error: Incorrect HTTP method for uri [/organizations/_doc/] and method [GET], allowed: [POST]

I think this is an issue with the latest version. I am also seeing this. @frikky Could you try to set the env variable SHUFFLE_BASE_IMAGE_NAME to frikky? It defaults to frikky/shuffle I think @alessio1983lx.

[alessio1983lx]

[WARNING] Failed to get current org : status: 405, error: Incorrect HTTP method for uri [/organizations/_doc/] and method [GET], allowed: [POST]

I think this is an issue with the latest version. I am also seeing this. @frikky Could you try to set the env variable SHUFFLE_BASE_IMAGE_NAME to frikky? It defaults to frikky/shuffle I think @alessio1983lx.

doing so i got a couldn't parse image name invalid reference format ....

[alessio1983lx]

Normal BackOff 2s (x2 over 33s) kubelet Back-off pulling image "docker.io/frikky:shuffle-tools_1.2.0"

using the environment var (as extra var for worker and uroboros) it messes the image name like that, should be docker.io/frikky/shuffle-tools:1.2.0 to work

[P4sca1]

Im currently working on the helm chart in https://github.com/Shuffle/Shuffle/pull/1893. I will try to fix default values, so that deployment works by default.