Chains controller uses `imagePullSecrets` for artifact upload, causing failure due to read-only registry access

[anithapriyanatarajan]

Expected Behavior

If the ServiceAccount used in the TaskRun or PipelineRun has both imagePullSecrets and secrets, chains controller is expected to use the mounted Secrets which would have push permission to the given registry rather than relying on imagePullSecrets which will have only READ permission to the targeted oci registry for signed artifact upload.

Actual Behavior

If the ServiceAccount used in the TaskRun or PipelineRun has both imagePullSecrets and secrets, chains controller is always using the imagePullSecrets to upload the signed artifacts resulting in AUTH error.

Steps to Reproduce the Problem

1. Install pipelines,chains in a kind cluster and complete config and execute a TaskRun following the script: https://gist.github.com/anithapriyanatarajan/ffa14efcc0274f7c22ec599df092ba21

2. verify that the annotation chains.tekton.dev/signed: "false" on TR after 3 retries

3. Remove the imagePullSecret attached the serviceAccount and start the TaskRun again. kubectl patch serviceaccount $SERVICE_ACCOUNT_NAME -p '{"imagePullSecrets": null}' -n $NAMESPACE

4. verify that the annotation chains.tekton.dev/signed: "true"

Additional Info

• Kubernetes version:

Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.0 

• Tekton Pipeline version:

pipeline version: 0.70.0

• Tekton Chain version:

chains version: 0.24.0

Relevance to another issue

https://github.com/tektoncd/chains/issues/700 - As part of this we have enabled usage of secrets from service Account.

https://github.com/lcarva/chains/blob/843c6b3477fee1154093d5a8660630f3ebdaef31/pkg/chains/storage/oci/oci.go#L58-L60 - here we do not seem to specify which secret to use for oci storage in options and hence as per the library both imagePullSecrets and Secrets are consolidated into a single slice and controller repeatedly fails always using the imagePullSecrets. This issue will be evident only if the oci registry is same for pull and push but the auth secret for pull is configured in imagePullSecrets and that for push is kept in secrets patched to the service account used by the chains controller

@jkhelil - Thankyou for detailed inputs regarding the library details that is causing this concern.

@lcarva - could you suggest a way forward for this issue? Is it fine to explicity include logic in oci backend storage to use only secrets but never imagePullSecrets for signed artifact upload.

[lcarva]

There has been enough time since we added support for secrets. I think it's reasonable to remove usage of imagePullSecrets now.

There are actually two sets of imagePullSecrets that are used.

The first case is the one mentioned in the issue. To remove such support, we need to change the underlying library here so it leverages an option to ignore the imagePullSecrets associated with the ServiceAccount, similar to the option to use secrets. Getting changes merged there has been... difficult. The Conforma/EC project has a fork which carries some fixes. We could consider using it and making the required changes there. This probably needs further discussion though.

The second case is usage of the imagePullSecrets that are defined in the Tekton resource itself, PodTemplate in a TaskRun for example. If we want to drop that support, we just have to remove this line.

[lcarva]

The workaround is to ensure that, for a given OCI repo, a single Secret is linked to the ServiceAccount, and that Secret has push access to the OCI repo. This one Secret could be listed in either imagePullSecrets or secrets of the ServiceAccount, but it's probably good practice to have it under secrets.