PipelineRun: Kaniko Docker Hub push fails due to invalid permissions

[tetra12]

Expected Behavior

Run pipeline successfully and deploy image to Docker Hub

Actual Behavior

Docker push fails with the UNAUTHORIZED behavior.

clone-build-push-rundqgdl-build-push-pod step-build-and-push error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated co
rrectly, and try again: checking push permission for "tampler/tekton-app:v0.0.1": POST https://index.docker.io/v2/tampler/tekton-app/blobs/uploads/: UNAUTHORIZED: authentica
tion required; [map[Action:pull Class: Name:tampler/tekton-app Type:repository] map[Action:push Class: Name:tampler/tekton-app Type:repository]]

The actual fix is the following:

    - name: docker-credentials
      secret:
        secretName: docker-creds-token
        items:                      
         - key: .dockerconfigjson
            path: config.json

Steps to Reproduce the Problem

1. Follow the CI tutorial
2. Create a secret with: kubectl create secret docker-registry docker-creds --from-file=.dockerconfigjson=/home/bku/.docker/config.json
3. Implement Service Account as:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: clone-build-push
spec:
  description: |
    This pipeline clones a git repo, builds a Docker image with Kaniko and
    pushes it to a registry
  params:
    - name: repo-url
      type: string
    - name: image-reference
      type: string
  workspaces:
    - name: shared-data
    - name: docker-credentials
  tasks:
    - name: fetch-source
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: shared-data
      params:
        - name: url
          value: $(params.repo-url)
    - name: build-push
      runAfter:
        - fetch-source
      taskRef:
        name: kaniko
      workspaces:
        - name: source
          workspace: shared-data
        - name: dockerconfig
          workspace: docker-credentials
      params:
        - name: IMAGE
          value: $(params.image-reference)
        - name: BUILDER_IMAGE
          value: gcr.io/kaniko-project/executor:latest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: docker-build-bot
  namespace: default
secrets:
  - name: docker-creds-token
imagePullSecrets:
  - name: docker-creds-token

5. Implement run as:

---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: clone-build-push-run
spec:
  serviceAccountName: docker-build-bot  <----- Doesn't make any difference and can be removed
  pipelineRef:
    name: clone-build-push
  podTemplate:
    securityContext:
      fsGroup: 65532
  workspaces:
    - name: shared-data
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
    - name: docker-credentials
      secret:
        secretName: docker-creds-token
#        items:                      <--------------- Uncomment to fix the issue
#         - key: .dockerconfigjson
#            path: config.json
  params:
    - name: repo-url
      value: https://github.com/google/docsy-example.git
    - name: image-reference
      value: tampler/tekton-app:v0.0.1

As per comments, ServiceAccountName: docker-build-bot doesn't make any difference and the Run fails with unauthorized. As a workaround, the commented lines must be uncommented.

However, as the best practice, it must work with a Service Account

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

Client version: 0.33.0
Pipeline version: v0.54.0
Dashboard version: v0.42.0

Using kind cluster kind v0.20.0 go1.21.4 linux/amd64

[vdemeester]

👋🏼 @tetra12. There is two things about this, but imo, none are really a bug of tektoncd/pipeline.

1. If you follow the tutorial, it clearly states to create the secret a "given" way (aka normal secret, base64 and using a key name config.json). Create a secret with: kubectl create secret docker-registry docker-creds --from-file=.dockerconfigjson=/home/bku/.docker/config.json is not "supported" by the tutorial (as the key of the secret is .dockerconfigjson. From that regard, we could update the tutorial to take into account that example as well (in the Container Registry Authentication part). This would become a "documentation enhancement" 👼🏼.
2. The kaniko task could be written to handle this type of secret without mapping the key (from .dockerconfigjson to config.json), like the openshift-pipelines buildah task does here. It would become a bug or feature enhancement on the kaniko Task in tektoncd/catalog.

I'll switch this issue to a "documentation" issue for now.

[aysegulozkaya]

We had the same problem. We have changed the secret yaml file like below. It works now. ***: base64 encoded version of the docker config file.

Previous:

kind: Secret
apiVersion: v1
metadata:
  name: tekton-pull-secret
  namespace: test-pipeline
data:
  .dockerconfigjson: ***
type: kubernetes.io/dockerconfigjson

New:

kind: Secret
apiVersion: v1
metadata:
  name: tekton-pull-secret
  namespace: test-pipeline
data:
  config.json:  ***
type: Opaque