Yet Another Authentication Problem

[dcarlet]

Actual behavior Kaniko fails to authenticate to local Artifactory registry using multiple different methods to generate the docker config json.

INFO[0000] Retrieving image manifest artifactory.local.domain/path/to/image:v1.0.0 
INFO[0000] Retrieving image artifactory.local.domain/path/to/image:v1.0.0 from registry artifactory.local.domain 
error building image: unable to complete operation after 0 attempts, last error: GET https://artifactory.local.domain/v2/path/to/image/manifests/v1.0.0: UNAUTHORIZED: The client does not have permission for manifest; map[manifest:path/to/image/v1.0.0/manifest.json]

Expected behavior Kaniko should correctly authenticate to container registries for both pull and push, and then correctly pull and build the image.

To Reproduce Steps to reproduce the behavior:

1. Set the /kaniko/.docker/config.json file auths in accordance with documentation for Artifactory ex:

  export ART_AUTH=$(echo -n ${ARTIFACTORY_CI_USER}:${ARTIFACTORY_CI_USER_ID_TOKEN} | base64 | tr -d '\n')
  echo "{\"auths\":{\"${ARTIFACTORY_REG}\":{\"auth\":\"${ART_AUTH}\"}}}" >> /kaniko/.docker/config.json

I also attempted it by generating the credentials using:

    echo "{\"auths\":{\"${ARTIFACTORY_REG}\":{\"auth\":\"$(printf "%s:%s" "${ARTIFACTORY_CI_USER}" "${ARTIFACTORY_CI_USER_ID_TOKEN}" | base64 | tr -d '\n')\"}}}" >> /kaniko/.docker/config.json

2. Attempt to build a dockerfile where the FROM image is specified in local Artifactory.

Additional Information

• Building the dockerfile locally works fine with both Podman and Docker.
• The docker config file in the kaniko container looks identical to my local podman auths.json file when generated in either fashion, and the base64 decode returns correct username/password. (Authenticated to the registry using ARTIFACTORY_CI_USER_ID_TOKEN as the password)
• Testing this outside of Gitlab CI also fails.
• Testing this outside of Gitlab CI by volume mounting in my current machine's podman auths.json file and the dockerfile also fails with the same error.
• Curling the manifest file succeeds:

curl --header "Authorization: Bearer ${ARTIFACTORY_CI_USER_ID_TOKEN}" -X GET https://artifactory.local.domain/v2/path/to/image/manifests/v3.4.11
<spew of manifest>

• The kaniko executor call from the pipeline (or local container attempts) is as follows:

/kaniko/executor --context "${CTX}" --dockerfile "${CTX}/Dockerfile" --destination "${CONTAINER_IMAGE}:${CONTAINER_TAG}" -v debug

• Dockerfile looks like:

FROM artifactory.local.domain/path/to/image:v1.0.0
COPY all-relevant-cas.crt /etc/ssl/certs/ca.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca.crt

• I additionally also tried with the --skip-tls-verify flags (various combinations) in case there was an SSL problem, and that didn't change anything.
• This auth method works just fine for the gitlab container registry (replacing the artifactory creds with $CI_REGISTRY, $CI_REGISTRY_USER, and $CI_REGISTRY_PASSWORD, obviously).
• Tested with versions v1.11.0 and v1.19.0

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [X]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[dcarlet]

I've also tried using the

{
  "username":"usernamehere"
  "password":"passwordhere"
}

format as well as a straight up copy of a dockerconfigjson from a k8s secret