error checking push permissions when pushing to insecure registry

[CeadeS]

Actual behavior When pushing to an insecure registry, process fails with the message:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again
: checking push permission for "adress:port/name:tag": Get https://server:port/v2/: http: server gave HTTP response to HTTPS client

Expected behavior Kaniko builds and pushes Image to registry. Still working at v0.9.0

To Reproduce Steps to reproduce the behavior:

config.yaml
...
args: ["--dockerfile=Dockerfile",    
            "--insecure-registry=YOUR_OWN_INSECURE_REGISTRY",
            "--destination=AN_EXISTING_DESTINATION",            
            "--insecure"]
...

[vijaySamanuri]

I hit with the same issue when using docker image by tag latest (updated 13 hours ago). I noticed issue with hostname resolution.

[cn-sunletian]

Look at Duplicate of #683

[priyawadhwa]

Hey @CeadeS could you try again with gcr.io/kaniko-project/executor: 51734fc3a33e04f113487853d118608ba6ff2b81 ? I'm wondering if #685 will fix your issue.

[tejal29]

@CeadeS and @CeadeS Can you please confirm?

[tejal29]

This is now fixed. thank you for your patience.

[pvgbabu]

@tejal29 , how did you fixed this issue, could you let us know

[just4est]

So??? How to fix this sh... ? There is no answer https://github.com/GoogleContainerTools/kaniko/issues/1179 to

[tejal29]

@just4est and @pvgbabu #685 should respect --insecure flag even if the --insecure-registries does not contain the destination image registry. #663 should skip tls verficaition for your insecure registry. Can you try --insecure and --skip-tls-verify ?

[tchellomello]

Hey @CeadeS could you try again with gcr.io/kaniko-project/executor: 51734fc3a33e04f113487853d118608ba6ff2b81 ? I'm wondering if #685 will fix your issue.

I was hitting the same issue and that image 51734fc3a33e04f113487853d118608ba6ff2b81 fixed the problem!

   steps:
     - name: build-and-push
       # https://github.com/GoogleContainerTools/kaniko/issues/702
       image: gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81
       #image: gcr.io/kaniko-project/executor:latest
       #image: gcr.io/kaniko-project/executor:debug
       # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
       env:
         - name: "DOCKER_CONFIG"
           value: "/tekton/home/.docker/"
       command:
         - /kaniko/executor
       args:
         - --dockerfile=$(params.pathToDockerFile)
         - --destination=$(params.registryURL)/$(params.imageOwner)/$(params.imageName):$(params.imageTag)
         - --context=$(params.pathToContext)
         - --insecure
         - --skip-tls-verify
         - --skip-tls-verify-pull
         - --insecure-pull

[luisgc93]

Just a heads up that the latest kaniko version (v1.7.0 ) seems to be causing this problem for my team: https://github.com/GoogleContainerTools/kaniko/releases

Downgrading to v1.6.0 fixed the problem.

[hamko]

@tejal29 Internally at Google, I see multiple high-tier GCP users report the same issue with the latest version. Could you find someone to take a look at this?

[peter-dough]

I saw some references in the release notes mentioning that some authentication binaries were removed for the slim executor images? Could it be that it was removed across the board?

[vitdolphin]

i have the same issue now on the pipeline that worked few days ago.

[austinpray-mixpanel]

I'm seeing this as well. And downgrading to 1.6.0 (gcr.io/kaniko-project/executor@sha256:0fae223f496525e31226cde473ec77ed15abfa8cbabff5ff5bf1c5268355bbb0) restored functionality.

[sigitisme]

yes using 1.6.0, solved my issue

[imjasonh]

In case it helps, here's a diff of the files in v1.6.0 and 1.7.0, using crane:

$ diff <(crane export gcr.io/kaniko-project/executor:v1.6.0-debug - | tar -tvf - | sort) <(crane export gcr.io/kaniko-project/executor:v1.7.0-debug - | tar -tvf - | sort)
1,4c1,2
< -rw-r--r--  0 root   root       25 Apr 26 12:54 kaniko/.config/gcloud/docker_credential_gcr_config.json
< -rw-r--r--  0 root   root      782 Apr 26 12:46 etc/nsswitch.conf
< -rw-r--r--  0 root   root   641760 Apr 26 12:47 kaniko/ssl/certs/ca-certificates.crt
< -rwxr-x---  0 root   root  6435607 Mar  5  2020 kaniko/docker-credential-gcr
---
> -rw-r--r--  0 root   root      782 Oct 19 22:29 etc/nsswitch.conf
> -rw-r--r--  0 root   root   641760 Oct 19 22:30 kaniko/ssl/certs/ca-certificates.crt
7,22c5,20
< -rwxr-xr-x  0 root   root 10223616 Apr 26 12:49 kaniko/docker-credential-ecr-login
< -rwxr-xr-x  0 root   root 13836465 Apr 26 12:50 kaniko/docker-credential-acr
< -rwxr-xr-x  0 root   root 33923072 Apr 26 12:54 kaniko/warmer
< -rwxr-xr-x  0 root   root 50630656 Apr 26 12:54 kaniko/executor
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 busybox/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 dev/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 etc/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/.config/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/.config/gcloud/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/.docker/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/ssl/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 kaniko/ssl/certs/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 proc/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 sys/
< drwxr-xr-x  0 root   root        0 Apr 26 12:54 workspace/
---
> -rwxr-xr-x  0 root   root  9829384 Oct 19 22:31 kaniko/docker-credential-gcr
> -rwxr-xr-x  0 root   root 10342400 Oct 19 22:32 kaniko/docker-credential-ecr-login
> -rwxr-xr-x  0 root   root 13836465 Oct 19 22:33 kaniko/docker-credential-acr
> -rwxr-xr-x  0 root   root 33959936 Oct 19 22:35 kaniko/warmer
> -rwxr-xr-x  0 root   root 50647040 Oct 19 22:35 kaniko/executor
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 bin/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 busybox/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 dev/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 etc/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/.docker/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/ssl/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 kaniko/ssl/certs/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 proc/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 sys/
> drwxr-xr-x  0 root   root        0 Oct 19 22:35 workspace/

It looks like kaniko/.config/gcloud/docker_credential_gcr_config.json has been removed since v1.6.0.

The contents of that file at v1.6.0:

$ crane export gcr.io/kaniko-project/executor:v1.6.0-debug - | tar -Oxf - kaniko/.config/gcloud/docker_credential_gcr_config.json
{"TokenSources":["env"]}

That seems to be caused by the the removal of this line:

RUN ["docker-credential-gcr", "config", "--token-source=env"]

...in this change, part of https://github.com/GoogleContainerTools/kaniko/pull/1780, 3 days ago.

I don't know if this is related, but it seems like a possible cause. That flag's behavior is documented here.

[imjasonh]

This comment on the other issue seems to indicate that re-adding docker-credential-gcr config --token-source=env might be (part of?) the fix, so that line in deploy/Dockerfile might be load-bearing.

I'm happy to send a PR to re-add this line, but I'd love to have some test to have more confidence that it is indeed the fix, and to prevent cases like this in the future.

[bygui86]

@imjasonh when do you release this fix?

[imjasonh]

Maintainers rolled back the 1.7.0 release to unbreak users, but I have no idea what the plan is for rolling it back forward or for finding or fixing the bug.

@tejal29 is there any update on this?

[bygui86]

@imjasonh thanks a lot

unfortunately I'm affected by #1667 so v1.6.0 is not suitable for me at the moment and I'm forced to use v1.6.0-debug :(

[edtshuma]

Hello @all. I am facing the same issue and I have tried most of the fixes to no avail :

1. I have tried using gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81 :

`before_script: - 'dotnet restore --packages $NUGET_PACKAGES_DIRECTORY'

build_job: tags: - xxxx only: - develop stage: build script: - dotnet build --configuration Release --no-restore

publish_job: tags: - xxxx only: - develop stage: publish artifacts: name: "$CI_COMMIT_SHA" paths: - ./$PUBLISH_DIR script: - dotnet publish ./src --configuration Release --output $(pwd)/$PUBLISH_DIR

docker_build_dev: tags: - xxxx image: name: gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81 entrypoint: [""] only: - develop stage: docker before_script: - echo "Docker build" script: - echo "${CI_PROJECT_DIR}" - cp ./src/Dockerfile /builds/xxx/xxx/xxx/Dockerfile - mkdir -p /kaniko/.docker - echo "{"auths":{"${CI_REGISTRY}":{"auth":"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')"}}}" > /kaniko/.docker/config.json - >- /kaniko/executor --context "${CI_PROJECT_DIR}" --insecure --skip-tls-verify --skip-tls-verify-pull --insecure-pull --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"`

2. I have also tried using gcr.io/kaniko-project/executor:v1.6.0-debug :

`before_script: - 'dotnet restore --packages $NUGET_PACKAGES_DIRECTORY'

build_job: tags: - xxxx only: - develop stage: build script: - dotnet build --configuration Release --no-restore

publish_job: tags: - xxxx only: - develop stage: publish artifacts: name: "$CI_COMMIT_SHA" paths: - ./$PUBLISH_DIR script: - dotnet publish ./src --configuration Release --output $(pwd)/$PUBLISH_DIR

docker_build_dev: tags: - xxxx image: name: gcr.io/kaniko-project/executor:v1.6.0-debug entrypoint: [""] only: - develop stage: docker before_script: - echo "Docker build" script: - echo "${CI_PROJECT_DIR}" - cp ./src/Dockerfile /builds/xxx/xxx/xxx/Dockerfile - mkdir -p /kaniko/.docker - echo "{"auths":{"${CI_REGISTRY}":{"auth":"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')"}}}" > /kaniko/.docker/config.json - >- /kaniko/executor --context "${CI_PROJECT_DIR}" --insecure --skip-tls-verify --skip-tls-verify-pull --insecure-pull --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG`

When using gcr.io/kaniko-project/executor:51734fc3a33e04f113487853d118608ba6ff2b81 the pipeline fails with error :

`[32;1mSkipping Git submodules setup[0;m section_end:1652535765:get_sources [0Ksection_start:1652535765:download_artifacts [0K[0K[36;1mDownloading artifacts[0;m[0;m [32;1mDownloading artifacts for publish_job (33475)...[0;m Downloading artifacts from coordinator... ok [0;m id[0;m=33475 responseStatus[0;m=200 OK token[xxxxxxxxxxx

section_end:1652535769:download_artifacts [0Ksection_start:1652535769:step_script [0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m section_end:1652535769:step_script [0Ksection_start:1652535769:cleanup_file_variables [0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m

section_end:1652539354:cleanup_file_variables [0K[31;1mERROR: Job failed: execution took longer than 1h0m0s seconds [0;m`

When using gcr.io/kaniko-project/executor:v1.6.0-debug the pipeline fails with error :

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: a repository name must be specified Cleaning up project directory and file based variables 00:00 ERROR: Job failed: command terminated with exit code 1

Is there something else obvious that I am missing?

[hades0932]

@Ed87 I have same error in minikube. If you use rancher desktop create cluster and use the same k8s version, maybe no problem. I don't know why。。。

[grid-dev]

Having the same error on EKS pushing cross account to ECR.

Using the following versions:

Kubernetes: 1.22
AMIs 1.22.6-20220526
Kaniko: gcr.io/kaniko-project/executor:v1.9.0@sha256:1f982af0b54be748221d9a35dcfa608660ab3d51229aa56bde5416f75aff7561

Error message

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[PLACEHOLDER.dkr.ecr.eu-central-1.amazonaws.com/PLACEHOLDER2/manager:latest](http://PLACEHOLDER.dkr.ecr.eu-central-1.amazonaws.com/PLACEHOLDER2/manager:latest)": Post "https://PLACEHOLDER.dkr.ecr.eu-central-1.amazonaws.com/v2/PLACEHOLDER2/manager/blobs/uploads/": EOF

[quyuancheng]

Can --insecure and --skip-tls-verify be configured at the same time? What is the impact?

[TK777777]

So how can we solve this problem in the end?

[hadrien-toma]

On my side I decided to use a secured registry because since a long time this issue was troubling my code and the workarounds were too costly to update along the releases...

[TK777777]

I have found the problem and need to set the variable of Gitlab as a non protected variable!!!!!