[BUG] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory

[Vyom-Yadav]

What did you do

• How was the cluster created?

$ k3d cluster create mycluster                            
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-mycluster'              
INFO[0000] Created image volume k3d-mycluster-images    
INFO[0000] Starting new tools node...                   
INFO[0000] Starting Node 'k3d-mycluster-tools'          
INFO[0001] Creating node 'k3d-mycluster-server-0'       
INFO[0001] Creating LoadBalancer 'k3d-mycluster-serverlb' 
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] HostIP: using network gateway 172.19.0.1 address 
INFO[0001] Starting cluster 'mycluster'                 
INFO[0001] Starting servers...                          
INFO[0001] Starting Node 'k3d-mycluster-server-0'       
INFO[0005] All agents already running.                  
INFO[0005] Starting helpers...                          
INFO[0005] Starting Node 'k3d-mycluster-serverlb'       
INFO[0011] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
INFO[0013] Cluster 'mycluster' created successfully!    
INFO[0013] You can now use it like this:                
kubectl cluster-info

• What did you do afterwards?
  • k3d commands?
  • docker commands?
  • OS operations (e.g. shutdown/reboot)?

Nothing, just kubectl get commands.

$ k get pods -A
E0408 16:01:20.071197  216996 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0408 16:01:20.072501  216996 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0408 16:01:20.073572  216996 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0408 16:01:20.074522  216996 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   helm-install-traefik-crd-tk874            0/1     ContainerCreating   0          80s
kube-system   helm-install-traefik-dc896                0/1     ContainerCreating   0          80s
kube-system   coredns-597584b69b-4rjnc                  0/1     ContainerCreating   0          79s
kube-system   local-path-provisioner-79f67d76f8-ngxkj   0/1     ContainerCreating   0          79s
kube-system   metrics-server-5f9f776df5-584hg           0/1     ContainerCreating   0          79s

On further investigation:

$ k describe pod -n kube-system metrics-server-5f9f776df5-584hg 
E0408 16:01:58.133390  217539 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0408 16:01:58.138322  217539 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0408 16:01:58.139287  217539 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0408 16:01:58.141336  217539 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Name:                 metrics-server-5f9f776df5-584hg
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      metrics-server
Node:                 k3d-mycluster-server-0/172.19.0.2
Start Time:           Sat, 08 Apr 2023 16:00:01 +0530
Labels:               k8s-app=metrics-server
                      pod-template-hash=5f9f776df5
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/metrics-server-5f9f776df5
Containers:
  metrics-server:
    Container ID:  
    Image:         rancher/mirrored-metrics-server:v0.6.2
    Image ID:      
    Port:          10250/TCP
    Host Port:     0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=10250
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --metric-resolution=15s
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get https://:https/livez delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4tm2f (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-4tm2f:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               116s  default-scheduler  Successfully assigned kube-system/metrics-server-5f9f776df5-584hg to k3d-mycluster-server-0
  Warning  FailedCreatePodSandBox  109s  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c57b12cbdf4dbb3820622ba349608e8b93b5b0253e5075b90a5aa9c4147c15b4": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  96s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a34d8af154cbec25ee180f173db338c0731e837bb57969471b606aacf760d8a2": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  82s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9a405bf6ad10864d8104added45e5bfbac84718c3e23337cbc7a59fca6c143e5": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  68s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ce6359d4c7e1d6798b018ec21ed657a749b581c9368a0bf98b972b02ded4618c": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  53s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8dc2d8edaf5d8dd88ad8ccefa6da466baa9af4baf6fdde7f4f4e74af3f300734": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  39s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4f42f982ee5a30d977c76b6dc5556908a5e2183375ff9cb00348f457c13e07cc": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  27s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d347b8c411a18880bb65b0af0af86aa4a04542965715aa007a419e040b038e29": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  13s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f7001b486fc75c49ea14abf51bbd87dc2f9c52bd53c26fbbb09355a77e6fba3c": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "XYZ": plugin type="flannel" failed (add): open /run/flannel/subnet.env: no such file or directory

What did you expect to happen

I expected the cluster to work.

Which OS & Architecture

• output of k3d runtime-info

Which version of k3d

$ k3d runtime-info                                             
arch: x86_64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: btrfs
infoname: fedora
name: docker
os: Fedora Linux 37 (Workstation Edition)
ostype: linux
version: 23.0.3

Which version of docker

$ docker version                                        
Client: Docker Engine - Community
 Version:           23.0.3
 API version:       1.42
 Go version:        go1.19.7
 Git commit:        3e7cbfd
 Built:             Tue Apr  4 22:10:33 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.3
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.7
  Git commit:       59118bf
  Built:            Tue Apr  4 22:07:34 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker info                
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 23.0.3
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.8-200.fc37.x86_64
 Operating System: Fedora Linux 37 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.47GiB
 Name: fedora
 ID: NIPB:2RVK:MPC3:YBZK:4DZH:XCIE:CLAC:3EEJ:JP6N:ME4I:WNXU:PTDU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[TomBraun02]

Hi, do you have a HTTP proxy on your network?

Please the output of: kubectl get pods -A

kubectl describe pod coredns-597584b69b-glmkk -n kube-system