Unable to sign image with kaniko when using --no-push arg

[R3DRUN3]

Expected Behavior

The image is signed.

Actual Behavior

The image is not signed.

Steps to Reproduce the Problem

I have the following tasks in a pipeline:

# Build OCI Image
 - name: kaniko-build
   runAfter: ["makefile-lint", "markdown-lint", "gitleaks", "pylint", "pytest", "docker-lint", "opa-dockerfile-validation", "helm-k8s-lint", "opa-helm-validation"]
   taskRef:
     name: kaniko
     kind: Task
   params:
     - name: IMAGE
       value: $(params.image-reference-prod)
     - name: EXTRA_ARGS
       value:
         - --skip-tls-verify
         - --insecure
         - --no-push
         - --tarPath=image.tar
   workspaces:
     - name: source
       workspace: shared-data
 # Image scan with trivy
 - name: trivy-scan-local-image
   runAfter: ["kaniko-build"]
   taskRef:
     name: trivy-scanner
     kind: Task
   params:
     - name: ARGS
       value: ["$(params.trivy_args_image_local[*])"]
     - name: IMAGE_PATH
       value: "/workspace/manifest-dir/image.tar"
   workspaces:
     - name: manifest-dir
       workspace: shared-data
 # Copy scanned image to production registry
 - name: skopeo-copy-to-production
   runAfter: ["trivy-scan-local-image"]
   taskRef:
     name: skopeo-copy
     kind: Task
   params:
     - name: srcImageURL
       value: "docker-archive:workspace/images-url/image.tar"
     - name: destImageURL
       value: "docker://$(params.image-reference-prod)"
     - name: srcTLSverify
       value: "false"
   workspaces:
     - name: images-url
       workspace: shared-data

So basically:

• build image with kaniko (local build with the --no-push and the --tarPath args)
• scan image with trivy
• copy image to remote docker registry

The pipeline work as intended but I am not able to sign the builded image (kaniko tasks) via Tekton chains. How can I sign the local .tar artifact produced by Kaniko (via tekton chains)?

Additional Info

• Kubernetes version:

  Output of kubectl version:

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

  Output of tkn version:

Client version: 0.30.1
Chains version: v0.16.0
Pipeline version: v0.47.0
Dashboard version: v0.35.0

TaskRun describe:

kubectl describe tr clone-build-push-run-fnnbf-kaniko-build

[ . . . ]

Events:
  Type     Reason           Age                    From                Message
  ----     ------           ----                   ----                -------
  Normal   FinalizerUpdate  10m                    taskrun-controller  Updated "clone-build-push-run-fnnbf-kaniko-build" finalizers
  Normal   Started          10m (x2 over 10m)      TaskRun             
  Normal   Pending          10m                    TaskRun             Pending
  Normal   Pending          10m                    TaskRun             pod status "Initialized":"False"; message: "containers with incomplete status: [prepare place-scripts working-dir-initializer]"
  Normal   Pending          10m                    TaskRun             pod status "Initialized":"False"; message: "containers with incomplete status: [place-scripts working-dir-initializer]"
  Normal   Pending          10m                    TaskRun             pod status "Initialized":"False"; message: "containers with incomplete status: [working-dir-initializer]"
  Normal   Pending          10m                    TaskRun             pod status "Ready":"False"; message: "containers with unready status: [step-build-and-push step-write-url]"
  Normal   Running          10m                    TaskRun             Not all Steps in the Task have finished executing
  Normal   Succeeded        9m46s                  TaskRun             All Steps have completed executing
  Warning  InternalError    9m40s (x5 over 9m44s)  taskrun-controller  1 error occurred:
           * getting signed image: entity not found in registry

[wlynch]

Sorry about the delayed response!

What I think is happening (assuming you're using the catalog tasks):

1. The kaniko task is outputting the IMAGE_URL/IMAGE_DIGEST results Chains is looking for.
2. Chains is picking this up, trying to sign. It tries to fetch the image (we do this to check if there's existing signatures on the image), fails creating the getting signed image: entity not found in registry
3. skopeo-copy runs which actually pushes the image to the registry, but the Task isn't outputting the results Chains is looking for so Chains never re-attempts to sign the image.

2 ways to solve this:

1. Modify the skopeo task to output the results that Chains is expecting
2. (probably the better long term solution) We follow https://github.com/sigstore/cosign/pull/2959 and allow Chains to sign images that don't actually exist in the registry yet.

[R3DRUN3]

Hi @wlynch, yes, I am using the catalog tasks. I agree with you that the second option is the best one.

Thank you!

[tekton-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.