Skaffold push always HTTPS despite insecure-registry flags

[robertgates55]

As part of a CI pipeline, I'm running a local registry, which is by default insecure: docker run -d --name registry.localhost -v local_registry:/var/lib/registry --rm -p 5000:5000 registry:2

I'm definitely able to resolve the registry (eg my /etc/hosts mappings are all good), however passing:

SKAFFOLD_DEFAULT_REPO="registry.localhost:5000"  skaffold build --insecure-registry=registry.localhost --insecure-registry=registry.localhost:5000 --cache-artifacts=false

Gives me:

couldn't build "blah": could not push image "registry.localhost:5000/blah:7d38d94d5b9-dirty": Get https://registry.localhost:5000/v2/: Service Unavailable

Expected behavior

At the very least, I'd have expected it to try and push to the http endpoint - http://registry.localhost:5000/v2

Actual behavior

It tries to push to https://registry.localhost:5000/v2

Information

• Skaffold version: v1.10.1
• Operating system: OSX

What might I be missing here? I've also tried the environment variable way of specifying the insecure registries. Does this only work with skaffold dev perhaps?

[tejal29]

@robertgates55 i was not able to reproduce your issue. I did the following

docker run -d --name registry.localhost -v local_registry:/var/lib/registry --rm -p 5000:5000 registry:2

This ran a registry

docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                                                      NAMES
7cb29a1f93d0        registry:2                            "/entrypoint.sh /etc…"   3 minutes ago       Up 3 minutes        0.0.0.0:5000->5000/tcp    

I ran skaffold build like you mentioned on latest skaffold 1.11.0 version.

skaffold build --insecure-registry=registry.localhost:5000 --cache-artifacts=false -d registry.localhost:5000

and the command was successful

Generating tags...
 - leeroy-web -> registry.localhost:5000/leeroy-web:v1.11.0-41-ge43a91d8a
 - leeroy-app -> registry.localhost:5000/leeroy-app:v1.11.0-41-ge43a91d8a
Found [minikube] context, using local docker daemon.
Building [leeroy-web]...
Sending build context to Docker daemon  3.072kB
Step 1/7 : FROM golang:1.12.9-alpine3.10 as builder
 ---> e0d646523991
...
Successfully built 85b7c5bc88d7
Successfully tagged registry.localhost:5000/leeroy-web:v1.11.0-41-ge43a91d8a
Building [leeroy-app]...
Sending build context to Docker daemon  3.072kB
...
Successfully tagged registry.localhost:5000/leeroy-app:v1.11.0-41-ge43a91d8a

[tejal29]

Setting local-cluster false, i can reproduce this error

skaffold config set --global local-cluster false
skaffold build --insecure-registry=registry.localhost:5000 --cache-artifacts=false -d registry.localhost:5000
couldn't build "leeroy-web": could not push image "registry.localhost:5000/leeroy-web:v1.11.0-41-ge43a91d8a": Get https://registry.localhost:5000/v2/: dial tcp: lookup registry.localhost on 192.168.65.1:53: no such host

[tejal29]

@dgageot I followed the code to figure out if there is a way to pass insecureRegistries to local docker daemon api client. The only way to found to load insecure registries was to create a /etc/docker/daemon.json file https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry

Is my investigation on the right track?

@robertgates55 do you have insecure registry configured like this?

[briandealwis]

@robertgates55 Apologies if this is a stupid question, but is registry.localhost a redacted host name, or did you actually set this up? I ask as I get the same Service Unavailable if I literally use registry.localhost:

$  docker run -d --name registry.localhost -v local_registry:/var/lib/registry --rm -p 5000:5000 registry:2                                 672baff8807ea0f6213ee461cf6e290e5569e916eb5b9360402bae0bf785ea56
$ ping registry.localhost
ping: cannot resolve registry.localhost: Unknown host
$ docker tag registry:2 registry.localhost:5000/registry
$ docker push registry.localhost:5000/registry
The push refers to repository [registry.localhost:5000/registry]
Get https://registry.localhost:5000/v2/: Service Unavailable

$ docker tag registry:2 localhost:5000/registry
$ docker push localhost:5000/registry
The push refers to repository [localhost:5000/registry]
73d61bf022fd: Pushed 
5bbc5831d696: Pushed 
d5974ddb5a45: Pushed 
f641ef7a37ad: Pushed 
d9ff549177a9: Pushed 
latest: digest: sha256:b1165286043f2745f45ea637873d61939bff6d9a59f76539d6228abf79f87774 size: 1363

[briandealwis]

@robertgates55 could you please confirm whether you actually have a hostname setup for registry.localhost? Or could you show the result of curl -i registry.localhost:5000?

[jrowinski3d]

Just like to add without creating a new issue but I too am having this issue. We have skaffold running in CI(Jenkins) where we are pushing our containers through a k8s service. Call it artifactory-artifactory:8081.

Regardless of setting environment variables, settings in the skaffold.yaml or passing via cli it continually sets itself to https

Successfully tagged artifactory-artifactory:8081/<redacted>:<tag>
The push refers to repository [artifactory-artifactory:8081/<redacted>]
Get https://artifactory-artifactory:8081/v2/: dial tcp: lookup artifactory-artifactory on 10.0.0.2:53: no such host

Inside our k8s cluster:

curl -i artifactory-artifactory:8081

HTTP/1.1 200 OK
Accept-Ranges: bytes
ETag: W/"878-1612379731000"
Last-Modified: Wed, 03 Feb 2021 19:15:31 GMT
Content-Type: text/html
Content-Length: 878
Date: Mon, 14 Jun 2021 20:14:33 GMT

Skaffold.yaml

build:
  insecureRegistries:
  - artifactory-artifactory:8081/<redacted_full_path>/
  - artifactory-artifactory:8081
  - artifactory-artifactory
  tagPolicy: &tag_policy
    gitCommit:
      variant: AbbrevCommitSha
  artifacts:
  - image: <image_redacted>
    context: <context_redacted>
    sync:
      infer:
        - '**/*.ts'
        - '**/*.tsx'
        - '**/*.css'
    docker:
      dockerfile: Dockerfile

[maximilize]

I have the same issue, the only working solution for me was to modify the docker config as described here: https://docs.docker.com/registry/insecure/

[briandealwis]

Cleaning up my mailbox and came across this issue.

The problem here is as discovered by @tejal29 above: Skaffold is unable to pass along its set of insecure registries as the Docker daemon API does not allow sending additional insecure-registries. See the Docker documentation for details on configuring the insecure registries.

I think the only thing we can do here is for Skaffold to check if the registry is marked as insecure and then error if that registry is not in the daemon's list of insecure registries (as returned by docker info):

ERROR[xxx] Docker daemon must be configured to treat 'registry.local' as an insecure-registry. See https://docs.docker.com/registry/insecure for instructions.

WDYT @tejal29?

---

To the OP @robertgates55:

At the very least, I'd have expected it to try and push to the http endpoint - http://registry.localhost:5000/v2

Generally it is unsafe to automatically downgrade connection attempts from https to http.

---

@jrowinski3d you're hitting a different error:

Get https://artifactory-artifactory:8081/v2/: dial tcp: lookup artifactory-artifactory on 10.0.0.2:53: no such host

Port 53 indicates a DNS issue: 10.0.0.2 is configured as your resolver and it is not valid. Note that Skaffold resolves hostnames from your local machine (or wherever you're running it from), and does not use the DNS set up that would be seen inside your cluster.

[jrowinski3d]

Hey @briandealwis , I am no longer working on that project. But that was running skaffold within a k8s cluster. But yes I would have tried to look more into the DNS issues, thought it would be able to resolve internally.

[r351574nc3]

I have a similar issue. I get this when running skaffold -v debug

DEBU[0025] push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true  subtask=-1 task=DevLoop
ERRO[0033] retrying net/http: TLS handshake timeout     
ERRO[0033] retrying net/http: TLS handshake timeout     
ERRO[0033] retrying net/http: TLS handshake timeout     
ERRO[0035] retrying net/http: TLS handshake timeout     
ERRO[0044] retrying net/http: TLS handshake timeout     
ERRO[0044] retrying net/http: TLS handshake timeout     
ERRO[0044] retrying net/http: TLS handshake timeout     
ERRO[0045] retrying net/http: TLS handshake timeout     
ERRO[0054] retrying net/http: TLS handshake timeout     
ERRO[0054] retrying net/http: TLS handshake timeout     
ERRO[0054] retrying net/http: TLS handshake timeout     
ERRO[0055] retrying net/http: TLS handshake timeout 

This is my ~/.skaffold/config

global:
  survey:
    last-prompted: "2023-11-02T11:11:12-07:00"
  collect-metrics: true
  update:
    last-prompted: "2023-11-04T19:41:34-07:00"
kubeContexts:
  - kube-context: rancher-desktop
    default-repo: localhost:5000
    insecure-registries:
      - localhost:5000