k3d icon indicating copy to clipboard operation
k3d copied to clipboard
verified publisher icon k3d-io

[BUG] K3d on Arch Linux with Rootless Podman - Failed to create cluster

Open tobru opened this issue 1 year ago • 2 comments

What did you do

k3d cluster create --verbose --trace, like described here: https://k3d.io/v5.6.3/usage/advanced/podman/#using-podman

What did you expect to happen

I expect to have a running k3s cluster on my local machine.

Screenshots or terminal output

DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/run/user/1000/podman/podman.sock Version:5.0.1 OSType:linux OS:arch Arch:amd64 CgroupVersion:2 CgroupDriver:systemd Filesystem:extfs InfoName:carbon} 
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  k3s-node-labels: []
  k3sargs: []
  ports: []
  registries:
    create: ""
  runtime-labels: []
  runtime-ulimits: []
  volumes: []
hostaliases: [] 
DEBU[0000] Configuration:
agents: 0
image: docker.io/rancher/k3s:v1.28.8-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    loadbalancer:
      configoverrides: []
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: ""
    gpurequest: ""
    hostpidmode: false
    serversmemory: ""
registries:
  config: ""
  use: []
servers: 1
subnet: ""
token: "" 
TRAC[0000] Trying to read config apiVersion='k3d.io/v1alpha5', kind='simple' 
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.28.8-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
========================== 
TRAC[0000] VolumeFilterMap: map[]                       
TRAC[0000] PortFilterMap: map[]                         
TRAC[0000] K3sNodeLabelFilterMap: map[]                 
TRAC[0000] RuntimeLabelFilterMap: map[]                 
TRAC[0000] EnvFilterMap: map[]                          
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha5} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:36067} Image:docker.io/rancher/k3s:v1.28.8-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[] Ulimits:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
========================== 
DEBU[0000] generated loadbalancer config:
ports:
  6443.tcp:
  - k3d-k3s-default-server-0
settings:
  workerConnections: 1024 
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:k3s-default Network:{Name:k3d-k3s-default ID: External:false IPAM:{IPPrefix:invalid Prefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0006028c0 0xc000602a80] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc00058e600 ServerLoadBalancer:0xc00004e880 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== ===== 
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created image volume k3d-k3s-default-images  
TRAC[0000] Using Registries: []                         
TRAC[0000] 
===== Creating Cluster =====

Runtime:
{}

Cluster:
&{Name:k3s-default Network:{Name:k3d-k3s-default ID:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 External:false IPAM:{IPPrefix:10.89.1.0/24 IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0006028c0 0xc000602a80] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc00058e600 ServerLoadBalancer:0xc00004e880 ImageVolume:k3d-k3s-default-images Volumes:[k3d-k3s-default-images]}

ClusterCreatOpts:
&{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 k3d.cluster.network.iprange:10.89.1.0/24] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}}

============================
         
TRAC[0000] Docker Machine not specified via DOCKER_MACHINE_NAME env var 
TRAC[0000] [Docker] Not using docker-machine            
DEBU[0000] [Docker] DockerHost: '' (unix:///run/user/1000/podman/podman.sock) 
INFO[0000] Starting new tools node...                   
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
TRAC[0000] Creating node from spec
&{Name:k3d-k3s-default-tools Role:noRole Image:ghcr.io/k3d-io/k3d-tools:5.6.3 Volumes:[k3d-k3s-default-images:/k3d/images /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.version:v5.6.3] RuntimeUlimits:[] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[host.k3d.internal:host-gateway] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[] K3dEntrypoint:false} 
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] [autofix cgroupsv2] cgroupVersion: 2         
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-default-tools Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[noop] Healthcheck:<nil> ArgsEscaped:false Image:ghcr.io/k3d-io/k3d-tools:5.6.3 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s-default k3d.role:noRole k3d.version:v5.6.3] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-default-images:/k3d/images /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[host.k3d.internal:host-gateway] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc0001dd26a} NetworkingConfig:{EndpointsConfig:map[k3d-k3s-default:0xc00018c1c0]}} 
DEBU[0000] Created container k3d-k3s-default-tools (ID: 77dae5d3e6ffe0828a491158f6174313dbe3b0aca70023d73a21289d7eac6ee0) 
DEBU[0000] Node k3d-k3s-default-tools Start Time: 2024-04-12 16:47:17.152557102 +0200 CEST m=+0.628482474 
TRAC[0000] Starting node 'k3d-k3s-default-tools'        
INFO[0000] Starting node 'k3d-k3s-default-tools'        
DEBU[0001] Truncated 2024-04-12 14:47:17.493392817 +0000 UTC to 2024-04-12 14:47:17 +0000 UTC 
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
TRAC[0001] Creating node from spec
&{Name:k3d-k3s-default-server-0 Role:server Image:docker.io/rancher/k3s:v1.28.8-k3s1 Volumes:[k3d-k3s-default-images:/k3d/images] Env:[K3S_TOKEN=UIuJqADxkZaaBTaKdgJr] Cmd:[] Args:[] Ports:map[] Restart:true Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:UIuJqADxkZaaBTaKdgJr k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.server.loadbalancer:k3d-k3s-default-serverlb] RuntimeUlimits:[] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:0xc00058e600} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[] K3dEntrypoint:false} 
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-default-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=UIuJqADxkZaaBTaKdgJr K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0 --tls-san k3d-k3s-default-serverlb] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.28.8-k3s1 Volumes:map[] WorkingDir: Entrypoint:[/bin/k3d-entrypoint.sh] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:UIuJqADxkZaaBTaKdgJr k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:36067 k3d.server.loadbalancer:k3d-k3s-default-serverlb k3d.version:v5.6.3] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-default-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc0004fd3ca} NetworkingConfig:{EndpointsConfig:map[k3d-k3s-default:0xc000286000]}} 
DEBU[0001] Created container k3d-k3s-default-server-0 (ID: 91d305fe530ff5216ed33b8fc281c480b4e09872d75daf4a3b7b41b9d4f66509) 
DEBU[0001] Created node 'k3d-k3s-default-server-0'      
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
TRAC[0001] Creating node from spec
&{Name:k3d-k3s-default-serverlb Role:loadbalancer Image:ghcr.io/k3d-io/k3d-proxy:5.6.3 Volumes:[k3d-k3s-default-images:/k3d/images] Env:[] Cmd:[] Args:[] Ports:map[6443:[{HostIP:0.0.0.0 HostPort:36067}]] Restart:true Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:UIuJqADxkZaaBTaKdgJr k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.role:loadbalancer k3d.server.loadbalancer:k3d-k3s-default-serverlb k3d.version:v5.6.3] RuntimeUlimits:[] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[{Stage:preStart Action:{Runtime:{} Content:[112 111 114 116 115 58 10 32 32 54 52 52 51 46 116 99 112 58 10 32 32 45 32 107 51 100 45 107 51 115 45 100 101 102 97 117 108 116 45 115 101 114 118 101 114 45 48 10 115 101 116 116 105 110 103 115 58 10 32 32 119 111 114 107 101 114 67 111 110 110 101 99 116 105 111 110 115 58 32 49 48 50 52 10] Dest:/etc/confd/values.yaml Mode:-rwxr--r-- Description:Write Loadbalancer Configuration}}] K3dEntrypoint:false} 
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-default-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443:{}] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:ghcr.io/k3d-io/k3d-proxy:5.6.3 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 k3d.cluster.network.iprange:10.89.1.0/24 k3d.cluster.token:UIuJqADxkZaaBTaKdgJr k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.role:loadbalancer k3d.server.loadbalancer:k3d-k3s-default-serverlb k3d.version:v5.6.3] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-default-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[6443:[{HostIP:0.0.0.0 HostPort:36067}]] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00040524a} NetworkingConfig:{EndpointsConfig:map[k3d-k3s-default:0xc00018c620]}} 
DEBU[0001] Created container k3d-k3s-default-serverlb (ID: b01c063bfdf017185c6bdf34825a5c5560d2bb2114aa8c3a2691ef95c9246ecf) 
DEBU[0001] Created loadbalancer 'k3d-k3s-default-serverlb' 
DEBU[0001] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
INFO[0001] Using the k3d-tools node to gather environment information 
TRAC[0001] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-k3s-default-tools 
DEBU[0001] no netlabel present on container /k3d-k3s-default-tools 
DEBU[0001] failed to get IP for container /k3d-k3s-default-tools as we couldn't find the cluster network 
DEBU[0001] Deleting node k3d-k3s-default-tools ...      
TRAC[0002] [Docker] Deleted Container k3d-k3s-default-tools 
DEBU[0002] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
TRAC[0002] GOOS: linux / Runtime OS: linux (arch)       
INFO[0002] HostIP: using network gateway 10.89.1.1 address 
INFO[0002] Starting cluster 'k3s-default'               
INFO[0002] Starting servers...                          
DEBU[0002] >>> enabling dns magic                       
DEBU[0002] >>> enabling cgroupsv2 magic                 
DEBU[0002] >>> enabling mounts magic                    
DEBU[0002] Node k3d-k3s-default-server-0 Start Time: 2024-04-12 16:47:19.145639076 +0200 CEST m=+2.621564389 
TRAC[0002] Node k3d-k3s-default-server-0: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 904 bytes to /bin/k3d-entrypoint.sh (mode -rwxr--r--): Write custom k3d entrypoint script (that powers the magic fixes) 
TRAC[0002] Node k3d-k3s-default-server-0: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 1469 bytes to /bin/k3d-entrypoint-dns.sh (mode -rwxr--r--): Write entrypoint script for DNS fix 
TRAC[0002] Node k3d-k3s-default-server-0: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 1325 bytes to /bin/k3d-entrypoint-cgroupv2.sh (mode -rwxr--r--): Write entrypoint script for CGroupV2 fix 
TRAC[0002] Node k3d-k3s-default-server-0: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 124 bytes to /bin/k3d-entrypoint-mounts.sh (mode -rwxr--r--): Write entrypoint script for mounts fix 
TRAC[0002] Starting node 'k3d-k3s-default-server-0'     
INFO[0002] Starting node 'k3d-k3s-default-server-0'     
DEBU[0003] Truncated 2024-04-12 14:47:19.77369923 +0000 UTC to 2024-04-12 14:47:19 +0000 UTC 
DEBU[0003] Waiting for node k3d-k3s-default-server-0 to get ready (Log: 'k3s is up and running') 
TRAC[0003] NodeWaitForLogMessage: Node 'k3d-k3s-default-server-0' waiting for log message 'k3s is up and running' since '2024-04-12 14:47:19 +0000 UTC' 
TRAC[0027] Non-fatal last log line in node k3d-k3s-default-server-0:  
ERRO[0028] Failed Cluster Start: Failed to start server k3d-k3s-default-server-0: Node k3d-k3s-default-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-k3s-default-server-0': stopped returning log lines: node k3d-k3s-default-server-0 is running=true in status=running 
ERRO[0028] Failed to create cluster >>> Rolling Back    
INFO[0028] Deleting cluster 'k3s-default'               
TRAC[0028] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-k3s-default-server-0 
TRAC[0028] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-k3s-default-serverlb 
DEBU[0028] Cluster Details: &{Name:k3s-default Network:{Name:k3d-k3s-default ID:c00e26b90e78ee9e88387fb636b60889af5d83d564682ab495dd543007866f10 External:false IPAM:{IPPrefix:10.89.1.0/24 IPsUsed:[] Managed:false} Members:[]} Token:UIuJqADxkZaaBTaKdgJr Nodes:[0xc0006028c0 0xc000602a80] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc00058e600 ServerLoadBalancer:0xc00004e880 ImageVolume:k3d-k3s-default-images Volumes:[k3d-k3s-default-images]} 
DEBU[0028] Deleting node k3d-k3s-default-serverlb ...   
TRAC[0028] [Docker] Deleted Container k3d-k3s-default-serverlb 
DEBU[0028] Deleting node k3d-k3s-default-server-0 ...   
TRAC[0028] [Docker] Deleted Container k3d-k3s-default-server-0 
INFO[0028] Deleting cluster network 'k3d-k3s-default'   
INFO[0028] Deleting 1 attached volumes...               
DEBU[0028] Deleting volume k3d-k3s-default-images...    
FATA[0029] Cluster creation FAILED, all changes have been rolled back!

And in the systemd journal I can see:

Apr 12 16:44:15 carbon systemd[2387]: Started /usr/lib/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Apr 12 16:44:15 carbon systemd[2387]: Started libcrun container.
Apr 12 16:44:15 carbon podman[1541622]: 2024-04-12 16:44:15.40727284 +0200 CEST m=+0.695139127 container init 62a328e466a785d402bb59c0c5b7bda2c88bfcf0bc2886ec3bb22512954176ce (image=docker.io/rancher/k3s:v1.28.8-k3s1, name=k3d-k3s-default-server-0, k3d.cluster.network=k3d-k3s-default, k3d.cluster.network.id=66c65ae4609457829837608f8c7af8d5ecf10b1b90411082fc7aba555cf1fc2b, k3d.version=v5.6.3, k3d.cluster.token=TgrpoTcaEDfjDfOUhqgK, k3d.server.api.port=36757, org.opencontainers.image.revision=653dd61aaa2d0ef8bd83ac4dbc6d150dde792efc, k3d.cluster.network.iprange=10.89.1.0/24, k3d.cluster.imageVolume=k3d-k3s-default-images, org.opencontainers.image.url=https://github.com/k3s-io/k3s, org.opencontainers.image.created=2024-03-26T00:49:54Z, k3d.server.api.host=0.0.0.0, app=k3d, k3d.cluster.network.external=false, k3d.cluster.url=https://k3d-k3s-default-server-0:6443, k3d.role=server, k3d.server.api.hostIP=0.0.0.0, org.opencontainers.image.source=https://github.com/k3s-io/k3s.git, k3d.server.loadbalancer=k3d-k3s-default-serverlb, k3d.cluster=k3s-default)
Apr 12 16:44:15 carbon podman[1541622]: 2024-04-12 16:44:15.415263156 +0200 CEST m=+0.703129388 container start 62a328e466a785d402bb59c0c5b7bda2c88bfcf0bc2886ec3bb22512954176ce (image=docker.io/rancher/k3s:v1.28.8-k3s1, name=k3d-k3s-default-server-0, k3d.cluster.network.id=66c65ae4609457829837608f8c7af8d5ecf10b1b90411082fc7aba555cf1fc2b, org.opencontainers.image.url=https://github.com/k3s-io/k3s, k3d.cluster.imageVolume=k3d-k3s-default-images, k3d.cluster.network.external=false, k3d.server.loadbalancer=k3d-k3s-default-serverlb, org.opencontainers.image.source=https://github.com/k3s-io/k3s.git, k3d.server.api.port=36757, k3d.cluster.network.iprange=10.89.1.0/24, org.opencontainers.image.revision=653dd61aaa2d0ef8bd83ac4dbc6d150dde792efc, org.opencontainers.image.created=2024-03-26T00:49:54Z, k3d.cluster=k3s-default, k3d.server.api.host=0.0.0.0, k3d.version=v5.6.3, k3d.cluster.url=https://k3d-k3s-default-server-0:6443, k3d.role=server, k3d.server.api.hostIP=0.0.0.0, app=k3d, k3d.cluster.network=k3d-k3s-default, k3d.cluster.token=TgrpoTcaEDfjDfOUhqgK)
Apr 12 16:44:15 carbon podman[1521032]: @ - - [12/Apr/2024:16:44:15 +0200] "GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22app%3Dk3d%22%3Atrue%2C%22k3d.cluster.imageVolume%3Dk3d-k3s-default-images%22%3Atrue%2C%22k3d.cluster.network.external%3Dfalse%22%3Atrue%2C%22k3d.cluster.network.id%3D66c65ae4609457829837608f8c7af8d5ecf10b1b90411082fc7aba555cf1fc2b%22%3Atrue%2C%22k3d.cluster.network.iprange%3D10.89.1.0%2F24%22%3Atrue%2C%22k3d.cluster.network%3Dk3d-k3s-default%22%3Atrue%2C%22k3d.cluster.token%3DTgrpoTcaEDfjDfOUhqgK%22%3Atrue%2C%22k3d.cluster.url%3Dhttps%3A%2F%2Fk3d-k3s-default-server-0%3A6443%22%3Atrue%2C%22k3d.cluster%3Dk3s-default%22%3Atrue%2C%22k3d.role%3Dserver%22%3Atrue%2C%22k3d.server.api.host%3D0.0.0.0%22%3Atrue%2C%22k3d.server.api.hostIP%3D0.0.0.0%22%3Atrue%2C%22k3d.server.api.port%3D36757%22%3Atrue%2C%22k3d.server.loadbalancer%3Dk3d-k3s-default-serverlb%22%3Atrue%2C%22k3d.version%3Dv5.6.3%22%3Atrue%7D%2C%22name%22%3A%7B%22%5E%2F%3F%28k3d-%29%3Fk3d-k3s-default-server-0%24%22%3Atrue%7D%7D HTTP/1.1" 200 3266 "" "Docker-Client/unknown-version (linux)"
Apr 12 16:44:15 carbon podman[1521032]: @ - - [12/Apr/2024:16:44:15 +0200] "HEAD /_ping HTTP/1.1" 200 0 "" "Docker-Client/unknown-version (linux)"
Apr 12 16:44:15 carbon podman[1521032]: @ - - [12/Apr/2024:16:44:15 +0200] "GET /v1.41/containers/62a328e466a785d402bb59c0c5b7bda2c88bfcf0bc2886ec3bb22512954176ce/json HTTP/1.1" 200 7673 "" "Docker-Client/unknown-version (linux)"
Apr 12 16:44:15 carbon conmon[1541842]: conmon 62a328e466a785d402bb <ninfo>: container 1541844 exited with status 1
Apr 12 16:44:15 carbon podman[1541874]: 2024-04-12 16:44:15.494220924 +0200 CEST m=+0.015207582 container died 62a328e466a785d402bb59c0c5b7bda2c88bfcf0bc2886ec3bb22512954176ce (image=docker.io/rancher/k3s:v1.28.8-k3s1, name=k3d-k3s-default-server-0, k3d.cluster.network.iprange=10.89.1.0/24, org.opencontainers.image.url=https://github.com/k3s-io/k3s, k3d.role=server, k3d.server.loadbalancer=k3d-k3s-default-serverlb, k3d.cluster=k3s-default, k3d.cluster.token=TgrpoTcaEDfjDfOUhqgK, k3d.version=v5.6.3, org.opencontainers.image.created=2024-03-26T00:49:54Z, app=k3d, org.opencontainers.image.source=https://github.com/k3s-io/k3s.git, k3d.cluster.imageVolume=k3d-k3s-default-images, k3d.server.api.host=0.0.0.0, k3d.cluster.network=k3d-k3s-default, k3d.server.api.port=36757, k3d.server.api.hostIP=0.0.0.0, org.opencontainers.image.revision=653dd61aaa2d0ef8bd83ac4dbc6d150dde792efc, k3d.cluster.network.external=false, k3d.cluster.network.id=66c65ae4609457829837608f8c7af8d5ecf10b1b90411082fc7aba555cf1fc2b, k3d.cluster.url=https://k3d-k3s-default-server-0:6443)
Apr 12 16:44:15 carbon podman[1541874]: 2024-04-12 16:44:15.534495929 +0200 CEST m=+0.055482639 container restart 62a328e466a785d402bb59c0c5b7bda2c88bfcf0bc2886ec3bb22512954176ce (image=docker.io/rancher/k3s:v1.28.8-k3s1, name=k3d-k3s-default-server-0, app=k3d, k3d.cluster.network.id=66c65ae4609457829837608f8c7af8d5ecf10b1b90411082fc7aba555cf1fc2b, org.opencontainers.image.url=https://github.com/k3s-io/k3s, k3d.role=server, k3d.version=v5.6.3, k3d.cluster.network.iprange=10.89.1.0/24, k3d.cluster.network=k3d-k3s-default, k3d.server.api.port=36757, org.opencontainers.image.source=https://github.com/k3s-io/k3s.git, k3d.cluster.imageVolume=k3d-k3s-default-images, k3d.cluster.url=https://k3d-k3s-default-server-0:6443, k3d.cluster.network.external=false, org.opencontainers.image.revision=653dd61aaa2d0ef8bd83ac4dbc6d150dde792efc, k3d.cluster.token=TgrpoTcaEDfjDfOUhqgK, org.opencontainers.image.created=2024-03-26T00:49:54Z, k3d.server.api.host=0.0.0.0, k3d.cluster=k3s-default, k3d.server.api.hostIP=0.0.0.0, k3d.server.loadbalancer=k3d-k3s-default-serverlb)
Apr 12 16:44:15 carbon aardvark-dns[1541782]: Received SIGHUP will refresh servers: 1
Apr 12 16:44:15 carbon aardvark-dns[1541782]: No configuration found stopping the sever

Which OS & Architecture

arch: amd64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /run/user/1000/podman/podman.sock
filesystem: extfs
infoname: carbon
name: docker
os: arch
ostype: linux
version: 5.0.1

Which version of k3d

k3d version v5.6.3
k3s version v1.28.8-k3s1 (default)

Which version of docker

Client:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.22.1
Git Commit:   946d055df324e4ed6c1e806b561af4740db4fea9-dirty
Built:        Tue Apr  2 22:02:08 2024
OS/Arch:      linux/amd64

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 95.39
    systemPercent: 1.38
    userPercent: 3.23
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2034
  hostname: carbon
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 984
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1008061865
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1008061865
  kernel: 6.8.4-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 4115079168
  memTotal: 33352388608
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.10.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.14.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_04_05.954589b-1
    version: |
      pasta 2024_04_05.954589b
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.3-1
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 15865249792
  swapTotal: 17179865088
  uptime: 43h 36m 52.00s (Approximately 1.79 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/tobru/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tobru/.local/share/containers/storage
  graphRootAllocated: 989315297280
  graphRootUsed: 318966857728
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 32
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/tobru/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.1
  Built: 1712088128
  BuiltTime: Tue Apr  2 22:02:08 2024
  GitCommit: 946d055df324e4ed6c1e806b561af4740db4fea9-dirty
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.1

tobru avatar Apr 12 '24 14:04 tobru

I think we're having the same issue as well. It works when downgrading to k3d 5.6.0. Possible cause: https://github.com/k3d-io/k3d/pull/1349

brandonchinn178 avatar Apr 15 '24 18:04 brandonchinn178

Same as https://github.com/k3d-io/k3d/issues/1439 ?

Cub0n avatar May 08 '24 10:05 Cub0n