k3d
k3d copied to clipboard
[BUG] Cluster creation fails on macOS and asks for docker login repeatedly
What did you do
Tried to create a cluster with 1 server and 2 agents.
- How was the cluster created?
k3d cluster create my-cluster --servers 1 --agents 2 --port 9080:80@loadbalancer --port 9443:443@loadbalancer --api-port 6443 --k3s-arg "--disable=traefik@server:*"
It fails with a docker failed to pull the image 'docker.io/rancher/k3s:v1.25.7-k3s1': Error response from daemon: Please run 'docker login'
error, even though I had previously done docker login
.
If I run a docker pull 'docker.io/rancher/k3s:v1.25.7-k3s1'
before the cluster creation, it works.
Screenshots or terminal output
$ docker login
Authenticating with existing credentials...
Login Succeeded
$ k3d cluster create my-cluster --servers 1 --agents 2 --port 9080:80@loadbalancer --port 9443:443@loadbalancer --api-port 6443 --k3s-arg "--disable=traefik@server:*" --verbose
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:24.0.5 OSType:linux OS:Docker Desktop Arch:aarch64 CgroupVersion:2 CgroupDriver:cgroupfs Filesystem:extfs InfoName:docker-desktop}
DEBU[0000] Additional CLI Configuration:
cli:
api-port: "6443"
env: []
k3s-node-labels: []
k3sargs:
- --disable=traefik@server:*
ports:
- 9080:80@loadbalancer
- 9443:443@loadbalancer
registries:
create: ""
runtime-labels: []
volumes: []
hostaliases: []
DEBU[0000] Configuration:
agents: 2
image: docker.io/rancher/k3s:v1.25.7-k3s1
network: ""
options:
k3d:
disableimagevolume: false
disableloadbalancer: false
disablerollback: false
loadbalancer:
configoverrides: []
timeout: 0s
wait: true
kubeconfig:
switchcurrentcontext: true
updatedefaultkubeconfig: true
runtime:
agentsmemory: ""
gpurequest: ""
hostpidmode: false
serversmemory: ""
registries:
config: ""
use: []
servers: 1
subnet: ""
token: ""
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:2 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.25.7-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:2 ExposeAPI:{Host: HostIP:0.0.0.0 HostPort:6443} Image:docker.io/rancher/k3s:v1.25.7-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[{Port:9080:80 NodeFilters:[loadbalancer]} {Port:9443:443 NodeFilters:[loadbalancer]}] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[{Arg:--disable=traefik NodeFilters:[server:*]}] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
INFO[0000] portmapping '9080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] portmapping '9443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
DEBU[0000] generated loadbalancer config:
ports:
80.tcp:
- k3d-my-cluster-server-0
- k3d-my-cluster-agent-0
- k3d-my-cluster-agent-1
443.tcp:
- k3d-my-cluster-server-0
- k3d-my-cluster-agent-0
- k3d-my-cluster-agent-1
6443.tcp:
- k3d-my-cluster-server-0
settings:
workerConnections: 1024
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:my-cluster Network:{Name:k3d-my-cluster ID: External:false IPAM:{IPPrefix:zero IPPrefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0x140004751e0 0x14000475380 0x14000475520 0x140004756c0] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0x14000399680 ServerLoadBalancer:0x140002f2c90 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server
INFO[0000] Prep: Network
DEBU[0000] Found network {Name:k3d-my-cluster ID:956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3 Created:2023-08-08 06:38:48.462306428 +0000 UTC Scope:local Driver:bridge EnableIPv6:false IPAM:{Driver:default Options:map[] Config:[{Subnet:172.18.0.0/16 IPRange: Gateway:172.18.0.1 AuxAddress:map[]}]} Internal:false Attachable:false Ingress:false ConfigFrom:{Network:} ConfigOnly:false Containers:map[] Options:map[com.docker.network.bridge.enable_ip_masquerade:true] Labels:map[app:k3d] Peers:[] Services:map[]}
INFO[0000] Re-using existing network 'k3d-my-cluster' (956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3)
INFO[0000] Created image volume k3d-my-cluster-images
INFO[0000] Starting new tools node...
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] [Docker] Local DfD: using 'host.docker.internal'
DEBU[0000] [Docker] wanted to use 'host.docker.internal' as docker host, but it's not resolvable locally: lookup host.docker.internal: no such host
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Detected CgroupV2, enabling custom entrypoint (disable by setting K3D_FIX_CGROUPV2=false)
DEBU[0000] Created container k3d-my-cluster-tools (ID: 114d7b45cef1f7231e510818ffff93b6c618b4b9c568bf9b0f43a0d0adda2ab3)
DEBU[0000] Node k3d-my-cluster-tools Start Time: 2023-08-08 08:43:30.454212 +0200 CEST m=+0.133891918
INFO[0000] Starting Node 'k3d-my-cluster-tools'
DEBU[0000] Truncated 2023-08-08 06:43:30.616273461 +0000 UTC to 2023-08-08 06:43:30 +0000 UTC
INFO[0001] Creating node 'k3d-my-cluster-server-0'
ERRO[0001] Failed Cluster Creation: failed setup of server/agent node k3d-my-cluster-server-0: failed to create node: runtime failed to create node 'k3d-my-cluster-server-0': failed to create container for node 'k3d-my-cluster-server-0': docker failed to pull image 'docker.io/rancher/k3s:v1.25.7-k3s1': docker failed to pull the image 'docker.io/rancher/k3s:v1.25.7-k3s1': Error response from daemon: Please run 'docker login'
ERRO[0001] Failed to create cluster >>> Rolling Back
INFO[0001] Deleting cluster 'my-cluster'
DEBU[0001] no netlabel present on container /k3d-my-cluster-tools
DEBU[0001] failed to get IP for container /k3d-my-cluster-tools as we couldn't find the cluster network
DEBU[0001] Cluster Details: &{Name:my-cluster Network:{Name:k3d-my-cluster ID:956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3 External:true IPAM:{IPPrefix:172.18.0.0/16 IPsUsed:[172.18.0.1] Managed:false} Members:[]} Token:UXlFIDdKpumFYrwrVMSi Nodes:[0x140004751e0 0x14000475380 0x14000475520 0x140004756c0 0x140000036c0] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0x14000399680 ServerLoadBalancer:0x140002f2c90 ImageVolume:k3d-my-cluster-images Volumes:[k3d-my-cluster-images]}
DEBU[0001] Deleting node k3d-my-cluster-serverlb ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-serverlb': Error response from daemon: No such container: k3d-my-cluster-serverlb
DEBU[0001] Deleting node k3d-my-cluster-server-0 ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-server-0': Error response from daemon: No such container: k3d-my-cluster-server-0
DEBU[0001] Deleting node k3d-my-cluster-agent-0 ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-agent-0': Error response from daemon: No such container: k3d-my-cluster-agent-0
DEBU[0001] Deleting node k3d-my-cluster-agent-1 ...
ERRO[0001] docker failed to remove the container 'k3d-my-cluster-agent-1': Error response from daemon: No such container: k3d-my-cluster-agent-1
DEBU[0001] Deleting node k3d-my-cluster-tools ...
DEBU[0001] Skip deletion of cluster network 'k3d-my-cluster' because it's managed externally
INFO[0001] Deleting 1 attached volumes...
DEBU[0001] Deleting volume k3d-my-cluster-images...
FATA[0001] Cluster creation FAILED, all changes have been rolled back!
$ docker pull docker.io/rancher/k3s:v1.25.7-k3s1
v1.25.7-k3s1: Pulling from rancher/k3s
8f802876bc40: Pull complete
91fe88c41772: Pull complete
Digest: sha256:aa02a6508ec2b8aeac839443f3b69b24e7789a8d9f0d9cb24f2adc2f26d5a60a
Status: Downloaded newer image for rancher/k3s:v1.25.7-k3s1
docker.io/rancher/k3s:v1.25.7-k3s1
$ k3d cluster create my-cluster --servers 1 --agents 2 --port 9080:80@loadbalancer --port 9443:443@loadbalancer --api-port 6443 --k3s-arg "--disable=traefik@server:*"
INFO[0000] portmapping '9080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] portmapping '9443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] Prep: Network
INFO[0000] Re-using existing network 'k3d-my-cluster' (956af13d8ad8b4381e192932424ab02eb5b3ff54a0fe3a1909c5a042868a23f3)
INFO[0000] Created image volume k3d-my-cluster-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-my-cluster-tools'
INFO[0001] Creating node 'k3d-my-cluster-server-0'
INFO[0001] Creating node 'k3d-my-cluster-agent-0'
INFO[0001] Creating node 'k3d-my-cluster-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-my-cluster-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] Starting new tools node...
INFO[0001] Starting Node 'k3d-my-cluster-tools'
INFO[0002] Starting cluster 'my-cluster'
INFO[0002] Starting servers...
INFO[0002] Starting Node 'k3d-my-cluster-server-0'
INFO[0006] Starting agents...
INFO[0006] Starting Node 'k3d-my-cluster-agent-1'
INFO[0006] Starting Node 'k3d-my-cluster-agent-0'
INFO[0010] Starting helpers...
INFO[0010] Starting Node 'k3d-my-cluster-serverlb'
INFO[0016] Injecting records for hostAliases (incl. host.k3d.internal) and for 5 network members into CoreDNS configmap...
INFO[0018] Cluster 'my-cluster' created successfully!
INFO[0018] You can now use it like this:
kubectl cluster-info
Which OS & Architecture
$ k3d runtime-info
arch: aarch64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 24.0.5
macOS Ventura 13.5
Which version of k3d
$ k3d version
k3d version v5.4.9
k3s version v1.25.7-k3s1 (default)
Which version of docker
$ docker version; docker info
Client:
Cloud integration: v1.0.35-desktop+001
Version: 24.0.5
API version: 1.43
Go version: go1.20.6
Git commit: ced0996
Built: Fri Jul 21 20:32:30 2023
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.22.0 (117440)
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.6
Git commit: a61e2b4
Built: Fri Jul 21 20:35:38 2023
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.21
GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc:
Version: 1.1.7
GitCommit: v1.1.7-0-g860f061
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Client:
Version: 24.0.5
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2-desktop.1
Path: /Users/spremraj/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.20.2-desktop.1
Path: /Users/spremraj/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/spremraj/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.20
Path: /Users/spremraj/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.6
Path: /Users/spremraj/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/spremraj/.docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /Users/spremraj/.docker/cli-plugins/docker-scan
scout: Command line tool for Docker Scout (Docker Inc.)
Version: 0.20.0
Path: /Users/spremraj/.docker/cli-plugins/docker-scout
Server:
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 5
Server Version: 24.0.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 5.15.49-linuxkit-pr
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 5
Total Memory: 10.7GiB
Name: docker-desktop
ID: c365d28b-7020-42f3-85ed-c833122e8809
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
Same issue in windows.
PS C:> k3d version k3d version v5.6.0 k3s version v1.27.4-k3s1 (default)
If I do docker pull before the cluster creation, it works.