Unauthorized when trying to sign image in gitlab-ci pipeline

[manuschillerdev]

Description I am trying to sign OCI-Images with cosign within a gitlab-ci pipeline. Auto-generating the keypair and the corresponding project variables worked fine. When I try to sign an image the request is not authorized, though:

Created fresh repository.
Checking out 3be77678 as 04_signed_images...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
$ mkdir -p "/tmp/.docker" && echo "${DOCKER_AUTH_CONFIG}" > "/tmp/.docker/config.json"
$ cosign sign --key "gitlab://$CI_PROJECT_ID" "$IMAGE_DESTINATION:$CI_COMMIT_SHORT_SHA"
WARNING: Image reference <REDACTED>/blueprints/gitlab-ci:3be77678 uses a tag, not a digest, to identify the image to sign.
    This can lead you to sign a different image than the intended one. Please use a
    digest (example.com/ubuntu@sha256:abc1[23](<REDACTED>/blueprints/gitlab-ci/-/jobs/70402#L23)...) rather than tag
    (example.com/ubuntu:latest) for the input to cosign. The ability to refer to
    images by tag will be removed in a future release.
Error: signing [<REDACTED>/blueprints/gitlab-ci:3be77678]: accessing entity: GET https://<REDACTED>/v2/blueprints/gitlab-ci/manifests/3be77678: UNAUTHORIZED: access to the requested resource is not authorized
main.go:74: error during command execution: signing [<REDACTED>/blueprints/gitlab-ci:3be77678]: accessing entity: GET https://<REDACTED>/blueprints/gitlab-ci/manifests/3be77678: UNAUTHORIZED: access to the requested resource is not authorized

My pipeline definition:

variables:
  DOCKER_AUTH_CONFIG: '{"auths": {"$DOCKER_REGISTRY_URL": {"auth": "$DOCKER_REGISTRY_AUTH_CONFIG"}}}'
  IMAGE_DESTINATION: "$DOCKER_REGISTRY_URL/$CI_PROJECT_PATH"

stages:
  - oci

oci:build:
  stage: oci
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "${DOCKER_AUTH_CONFIG}" > /kaniko/.docker/config.json
    - /kaniko/executor
      --cleanup
      --cache
      --cache-repo "IMAGE_DESTINATION_cache"
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination "$IMAGE_DESTINATION:$CI_COMMIT_SHORT_SHA"

oci:sign:
  stage: oci
  needs:
    - oci:build
  variables:
    GITLAB_HOST: "https://<redacted>"
   # edit: just for testing - should oviously not be set in .gitlab-ci.yml 
    COSIGN_PASSWORD: "justasecrettest"
    DOCKER_CONFIG: "/tmp/.docker/config.json"
  image: "bitnami/cosign:latest"
  script:
    # note: kaniko uses the exact same configuration
    # the current user is not allowed to write to ~/.docker/config.json or /.docker/config.json, so I tried
    # to get around this by using `DOCKER_CONFIG` and a custom path `/tmp/.docker/config.json`
    - mkdir -p "/tmp/.docker" && echo "${DOCKER_AUTH_CONFIG}" > "/tmp/.docker/config.json"
    # this step suceeded in previous run, but needs to be conditional from now on since secret variables 
    # must not be recreated when they already exist
    #  - cosign generate-key-pair gitlab://$CI_PROJECT_ID
    - cosign sign --key "gitlab://$CI_PROJECT_ID" "$IMAGE_DESTINATION:$CI_COMMIT_SHORT_SHA"
    - cosign verify --key "gitlab://$CI_PROJECT_ID" "$IMAGE_DESTINATION:$CI_COMMIT_SHORT_SHA"

Version

GitVersion:    v2.0.2
GitCommit:     871448050b924a7946ebe47678f23aae09ef432d
GitTreeState:  clean
BuildDate:     2023-04-24T17:31:42Z
GoVersion:     go1.19.9

[cpanato]

looks like this pipeline does not have access to the secret. not sure 100%

also you should not expose the cosign password

[manuschillerdev]

after some debugging the problem seems two-fold:

1. cosign does not use DOCKER_CONFIG for locating the .docker/config.json file. This is not documented anywhere, I only found the option in https://github.com/sigstore/cosign/issues/587. So I am not sure if this is a bug, or just not intended to work?
2. the bitnami/cosign Image does not have a writable folder /.docker folder (see: https://github.com/bitnami/containers/issues/28690#issuecomment-1493876353). I'll open a PR for that on the bitnami/containers repo

[znewman01]

Thanks for the detailed debugging!

cosign does not use DOCKER_CONFIG for locating the .docker/config.json file. This is not documented anywhere, I only found the option in https://github.com/sigstore/cosign/issues/587. So I am not sure if this is a bug, or just not intended to work?

I believe https://github.com/sigstore/cosign/issues/587 says that cosign does respect $DOCKER_CONFIG. It's possible that this broke since then. But I believe we intend to respect it. Given that, I think we'd take a PR to fix it (CC @jonjohnsonjr to confirm).

[nontster]

By using docker image bitnami/cosign:2.0.2-debian-11-r10 I can solve the problem by put DOCKER_AUTH_CONFIG into /.docker/config.json

Using ${CI_PROJECT_DIR }/.docker/config.json or any custom location with DOCKER_CONFIG not working for me.

My pipeline code snippets,

variables:
  IMAGE_DESTINATION: "${HARBOR_URL}/${CI_PROJECT_ROOT_NAMESPACE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}"

docker:build:
  stage: build
  image: 
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"${HARBOR_URL}\":{\"auth\":\"$(printf "%s:%s" "${HARBOR_USER}" "${HARBOR_SECRET}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - cat /kaniko/.docker/config.json
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination ${IMAGE_DESTINATION}

docker:sign:
  stage: post-build
  needs: ["docker:build"]
  image: bitnami/cosign:2.0.2-debian-11-r10
  script:
    - echo "{\"auths\":{\"${HARBOR_URL}\":{\"auth\":\"$(printf "%s:%s" "${HARBOR_USER}" "${HARBOR_SECRET}" | base64 | tr -d '\n')\"}}}" > /.docker/config.json
    - cosign -d sign -y --tlog-upload=false --key awskms:///${KMS_COSIGN_KEY_ARN} ${IMAGE_DESTINATION}