k3d
k3d copied to clipboard
k3d-io
[BUG] Server's `Node` object is never created
What did you do
-
How was the cluster created?
k3d cluster create foo
-
What did you do afterwards?
- Tried to run a Pod (
kubectl run foo --image=busybox --rm -it) and noticed that it seemed to hang on startup - Inspected the Nod (
kubectl describe pod foo) and noted that it was never scheduled to a Node - Tried to find the Node (
kubectl get nodes) and noted that no Node objects exist
- Tried to run a Pod (
What did you expect to happen
- The
Podshould run and give me a shell. - The server node should be listed as a schedulable
Node.
Screenshots or terminal output
kubectl run/describe/get
$ kubectl run foo --image=busybox --rm -it&
$ kubectl describe pod foo
Name: foo
Namespace: default
Priority: 0
Node: <none>
Labels: run=foo
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
foo:
Image: busybox
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g8gfd (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-g8gfd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3s default-scheduler no nodes available to schedule pods
$ kubectl get nodes
No resources found
Curiously, the server container logs keep complaining about being unable to find runtime.v1alpha2.RuntimeService:
server container logs
time="2022-02-11T11:22:29Z" level=info msg="Starting k3s v1.22.2+k3s2 (3f5774b4)"
time="2022-02-11T11:22:29Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2022-02-11T11:22:29Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2022-02-11T11:22:29Z" level=info msg="Database tables and indexes are up to date"
time="2022-02-11T11:22:29Z" level=info msg="Kine available at unix://kine.sock"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:29 +0000 UTC"
time="2022-02-11T11:22:29Z" level=info msg="Active TLS secret (ver=) (count 10): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.29.0.3:172.29.0.3 listener.cattle.io/cn-k3d-foo-server-0:k3d-foo-server-0 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=FB473CA2A839175D300220BA26E34D832FC636FF]"
time="2022-02-11T11:22:29Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0211 11:22:29.536471 7 server.go:581] external host was not specified, using 172.29.0.3
I0211 11:22:29.536705 7 server.go:175] Version: v1.22.2+k3s2
I0211 11:22:29.768202 7 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0211 11:22:29.768873 7 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0211 11:22:29.768881 7 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0211 11:22:29.769595 7 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0211 11:22:29.769602 7 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0211 11:22:29.779471 7 genericapiserver.go:455] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0211 11:22:29.780027 7 instance.go:278] Using reconciler: lease
I0211 11:22:29.817694 7 rest.go:130] the default service ipfamily for this cluster is: IPv4
W0211 11:22:30.021452 7 genericapiserver.go:455] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.022708 7 genericapiserver.go:455] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.032456 7 genericapiserver.go:455] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.033970 7 genericapiserver.go:455] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.038492 7 genericapiserver.go:455] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.040597 7 genericapiserver.go:455] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0211 11:22:30.045087 7 genericapiserver.go:455] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.045096 7 genericapiserver.go:455] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0211 11:22:30.046159 7 genericapiserver.go:455] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0211 11:22:30.046165 7 genericapiserver.go:455] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0211 11:22:30.049073 7 genericapiserver.go:455] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0211 11:22:30.050559 7 genericapiserver.go:455] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0211 11:22:30.053615 7 genericapiserver.go:455] Skipping API apps/v1beta2 because it has no resources.
W0211 11:22:30.053622 7 genericapiserver.go:455] Skipping API apps/v1beta1 because it has no resources.
W0211 11:22:30.054959 7 genericapiserver.go:455] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
I0211 11:22:30.057739 7 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0211 11:22:30.057748 7 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0211 11:22:30.062189 7 genericapiserver.go:455] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
time="2022-02-11T11:22:30Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2022-02-11T11:22:30Z" level=info msg="Waiting for API server to become available"
time="2022-02-11T11:22:30Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
time="2022-02-11T11:22:30Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
time="2022-02-11T11:22:30Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2022-02-11T11:22:30Z" level=info msg="To join node to cluster: k3s agent -s https://172.29.0.3:6443 -t ${NODE_TOKEN}"
time="2022-02-11T11:22:30Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2022-02-11T11:22:30Z" level=info msg="Run: k3s kubectl"
time="2022-02-11T11:22:30Z" level=info msg="Cluster-Http-Server 2022/02/11 11:22:30 http: TLS handshake error from 127.0.0.1:59414: remote error: tls: bad certificate"
time="2022-02-11T11:22:30Z" level=info msg="Cluster-Http-Server 2022/02/11 11:22:30 http: TLS handshake error from 127.0.0.1:59420: remote error: tls: bad certificate"
time="2022-02-11T11:22:30Z" level=info msg="certificate CN=k3d-foo-server-0 signed by CN=k3s-server-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:30 +0000 UTC"
time="2022-02-11T11:22:30Z" level=info msg="certificate CN=system:node:k3d-foo-server-0,O=system:nodes signed by CN=k3s-client-ca@1644578549: notBefore=2022-02-11 11:22:29 +0000 UTC notAfter=2023-02-11 11:22:30 +0000 UTC"
time="2022-02-11T11:22:30Z" level=info msg="Module overlay was already loaded"
time="2022-02-11T11:22:30Z" level=info msg="Module nf_conntrack was already loaded"
time="2022-02-11T11:22:30Z" level=info msg="Module br_netfilter was already loaded"
time="2022-02-11T11:22:30Z" level=info msg="Module iptable_nat was already loaded"
time="2022-02-11T11:22:30Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 786432"
time="2022-02-11T11:22:30Z" level=error msg="Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied"
time="2022-02-11T11:22:30Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
time="2022-02-11T11:22:30Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
time="2022-02-11T11:22:30Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2022-02-11T11:22:30Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2022-02-11T11:22:31Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
I0211 11:22:31.308927 7 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
I0211 11:22:31.308927 7 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
I0211 11:22:31.309097 7 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
I0211 11:22:31.309304 7 secure_serving.go:266] Serving securely on 127.0.0.1:6444
I0211 11:22:31.309341 7 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0211 11:22:31.309387 7 available_controller.go:491] Starting AvailableConditionController
I0211 11:22:31.309395 7 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0211 11:22:31.309425 7 apf_controller.go:299] Starting API Priority and Fairness config controller
I0211 11:22:31.309473 7 naming_controller.go:291] Starting NamingConditionController
I0211 11:22:31.309495 7 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0211 11:22:31.309500 7 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0211 11:22:31.309530 7 establishing_controller.go:76] Starting EstablishingController
I0211 11:22:31.309534 7 autoregister_controller.go:141] Starting autoregister controller
I0211 11:22:31.309537 7 crd_finalizer.go:266] Starting CRDFinalizer
I0211 11:22:31.309543 7 cache.go:32] Waiting for caches to sync for autoregister controller
I0211 11:22:31.309480 7 customresource_discovery_controller.go:209] Starting DiscoveryController
I0211 11:22:31.309554 7 crdregistration_controller.go:111] Starting crd-autoregister controller
I0211 11:22:31.309559 7 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I0211 11:22:31.309724 7 controller.go:83] Starting OpenAPI AggregationController
I0211 11:22:31.309763 7 controller.go:85] Starting OpenAPI controller
I0211 11:22:31.309807 7 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0211 11:22:31.309817 7 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0211 11:22:31.309819 7 dynamic_serving_content.go:129] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key"
I0211 11:22:31.310214 7 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0211 11:22:31.310222 7 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0211 11:22:31.310257 7 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
I0211 11:22:31.310285 7 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
I0211 11:22:31.317094 7 controller.go:611] quota admission added evaluator for: namespaces
E0211 11:22:31.322062 7 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time
E0211 11:22:31.322869 7 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.29.0.3, ResourceVersion: 0, AdditionalErrorMsg:
I0211 11:22:31.368264 7 shared_informer.go:247] Caches are synced for node_authorizer
I0211 11:22:31.410153 7 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0211 11:22:31.410177 7 apf_controller.go:304] Running API Priority and Fairness config worker
I0211 11:22:31.410201 7 cache.go:39] Caches are synced for AvailableConditionController controller
I0211 11:22:31.410216 7 shared_informer.go:247] Caches are synced for crd-autoregister
I0211 11:22:31.410225 7 cache.go:39] Caches are synced for autoregister controller
I0211 11:22:31.410315 7 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
time="2022-02-11T11:22:32Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
I0211 11:22:32.309478 7 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0211 11:22:32.309491 7 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0211 11:22:32.313165 7 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
I0211 11:22:32.315773 7 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0211 11:22:32.315783 7 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0211 11:22:32.604198 7 controller.go:611] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0211 11:22:32.622982 7 controller.go:611] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0211 11:22:32.743176 7 lease.go:233] Resetting endpoints for master service "kubernetes" to [172.29.0.3]
I0211 11:22:32.743951 7 controller.go:611] quota admission added evaluator for: endpoints
I0211 11:22:32.747078 7 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
time="2022-02-11T11:22:33Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:33Z" level=info msg="Kube API server is now running"
time="2022-02-11T11:22:33Z" level=info msg="k3s is up and running"
time="2022-02-11T11:22:33Z" level=info msg="Waiting for cloud-controller-manager privileges to become available"
time="2022-02-11T11:22:33Z" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2022-02-11T11:22:33Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2022-02-11T11:22:33Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
time="2022-02-11T11:22:33Z" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
I0211 11:22:33.528947 7 serving.go:354] Generated self-signed cert in-memory
I0211 11:22:33.591165 7 serving.go:354] Generated self-signed cert in-memory
I0211 11:22:33.784159 7 controllermanager.go:186] Version: v1.22.2+k3s2
I0211 11:22:33.786204 7 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0211 11:22:33.786217 7 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0211 11:22:33.786223 7 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0211 11:22:33.786226 7 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0211 11:22:33.786233 7 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0211 11:22:33.786240 7 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0211 11:22:33.786555 7 secure_serving.go:200] Serving securely on 127.0.0.1:10257
I0211 11:22:33.786579 7 tlsconfig.go:240] "Starting DynamicServingCertificateController"
time="2022-02-11T11:22:33Z" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2022-02-11T11:22:33Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-10.3.0.tgz"
time="2022-02-11T11:22:33Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-10.3.0.tgz"
time="2022-02-11T11:22:33Z" level=info msg="Failed to get existing traefik HelmChart" error="helmcharts.helm.cattle.io \"traefik\" not found"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
time="2022-02-11T11:22:33Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
I0211 11:22:33.886390 7 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0211 11:22:33.886410 7 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0211 11:22:33.886392 7 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0211 11:22:33.896392 7 shared_informer.go:240] Waiting for caches to sync for tokens
I0211 11:22:33.903215 7 controller.go:611] quota admission added evaluator for: serviceaccounts
I0211 11:22:33.905009 7 controllermanager.go:577] Started "persistentvolume-expander"
I0211 11:22:33.905115 7 expand_controller.go:327] Starting expand controller
I0211 11:22:33.905122 7 shared_informer.go:240] Waiting for caches to sync for expand
I0211 11:22:33.910023 7 controllermanager.go:577] Started "daemonset"
W0211 11:22:33.910034 7 controllermanager.go:556] "bootstrapsigner" is disabled
W0211 11:22:33.910038 7 controllermanager.go:556] "route" is disabled
I0211 11:22:33.910093 7 daemon_controller.go:284] Starting daemon sets controller
I0211 11:22:33.910098 7 shared_informer.go:240] Waiting for caches to sync for daemon sets
I0211 11:22:33.915339 7 controllermanager.go:577] Started "pvc-protection"
I0211 11:22:33.915367 7 pvc_protection_controller.go:110] "Starting PVC protection controller"
I0211 11:22:33.915375 7 shared_informer.go:240] Waiting for caches to sync for PVC protection
I0211 11:22:33.921465 7 controllermanager.go:577] Started "endpointslicemirroring"
I0211 11:22:33.921584 7 endpointslicemirroring_controller.go:212] Starting EndpointSliceMirroring controller
I0211 11:22:33.921594 7 shared_informer.go:240] Waiting for caches to sync for endpoint_slice_mirroring
W0211 11:22:33.935352 7 authorization.go:47] Authorization is disabled
W0211 11:22:33.935365 7 authentication.go:47] Authentication is disabled
I0211 11:22:33.935374 7 deprecated_insecure_serving.go:54] Serving healthz insecurely on [::]:10251
I0211 11:22:33.936433 7 controllermanager.go:577] Started "namespace"
I0211 11:22:33.936505 7 namespace_controller.go:200] Starting namespace controller
I0211 11:22:33.936518 7 shared_informer.go:240] Waiting for caches to sync for namespace
I0211 11:22:33.937462 7 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0211 11:22:33.937468 7 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0211 11:22:33.937484 7 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0211 11:22:33.937490 7 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0211 11:22:33.937503 7 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0211 11:22:33.937511 7 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0211 11:22:33.937773 7 secure_serving.go:200] Serving securely on 127.0.0.1:10259
I0211 11:22:33.937792 7 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0211 11:22:33.941922 7 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-serving"
I0211 11:22:33.941934 7 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-serving
I0211 11:22:33.941963 7 dynamic_serving_content.go:129] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/server-ca.crt::/var/lib/rancher/k3s/server/tls/server-ca.key"
I0211 11:22:33.942053 7 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-client"
I0211 11:22:33.942059 7 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-client
I0211 11:22:33.942083 7 dynamic_serving_content.go:129] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key"
I0211 11:22:33.942133 7 certificate_controller.go:118] Starting certificate controller "csrsigning-kube-apiserver-client"
I0211 11:22:33.942140 7 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client
I0211 11:22:33.942161 7 dynamic_serving_content.go:129] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key"
I0211 11:22:33.942253 7 controllermanager.go:577] Started "csrsigning"
I0211 11:22:33.942312 7 certificate_controller.go:118] Starting certificate controller "csrsigning-legacy-unknown"
I0211 11:22:33.942317 7 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-legacy-unknown
I0211 11:22:33.942331 7 dynamic_serving_content.go:129] "Starting controller" name="csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key"
I0211 11:22:33.948057 7 controllermanager.go:577] Started "serviceaccount"
I0211 11:22:33.948137 7 serviceaccounts_controller.go:117] Starting service account controller
I0211 11:22:33.948147 7 shared_informer.go:240] Waiting for caches to sync for service account
I0211 11:22:33.954029 7 controllermanager.go:577] Started "job"
I0211 11:22:33.954133 7 job_controller.go:172] Starting job controller
I0211 11:22:33.954140 7 shared_informer.go:240] Waiting for caches to sync for job
I0211 11:22:33.959702 7 controllermanager.go:577] Started "replicaset"
I0211 11:22:33.959800 7 replica_set.go:186] Starting replicaset controller
I0211 11:22:33.959812 7 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
I0211 11:22:33.961981 7 controllermanager.go:577] Started "csrapproving"
I0211 11:22:33.962067 7 certificate_controller.go:118] Starting certificate controller "csrapproving"
I0211 11:22:33.962077 7 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving
I0211 11:22:33.964015 7 controllermanager.go:577] Started "csrcleaner"
I0211 11:22:33.964031 7 cleaner.go:82] Starting CSR cleaner controller
time="2022-02-11T11:22:33Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
I0211 11:22:33.996771 7 shared_informer.go:247] Caches are synced for tokens
time="2022-02-11T11:22:34Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:34Z" level=info msg="Cluster dns configmap has been set successfully"
I0211 11:22:34.037497 7 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0211 11:22:34.037572 7 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0211 11:22:34.037580 7 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0211 11:22:34.101993 7 controllermanager.go:577] Started "endpoint"
I0211 11:22:34.102046 7 endpoints_controller.go:195] Starting endpoint controller
I0211 11:22:34.102055 7 shared_informer.go:240] Waiting for caches to sync for endpoint
time="2022-02-11T11:22:34Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting apps/v1, Kind=Deployment controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=Node controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=Service controller"
time="2022-02-11T11:22:34Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
I0211 11:22:34.250208 7 controllermanager.go:577] Started "endpointslice"
I0211 11:22:34.250252 7 endpointslice_controller.go:257] Starting endpoint slice controller
I0211 11:22:34.250260 7 shared_informer.go:240] Waiting for caches to sync for endpoint_slice
I0211 11:22:34.557363 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for jobs.batch
I0211 11:22:34.557417 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
I0211 11:22:34.557446 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0211 11:22:34.557468 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0211 11:22:34.557534 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for serviceaccounts
I0211 11:22:34.557572 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for deployments.apps
I0211 11:22:34.557642 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for replicasets.apps
I0211 11:22:34.557662 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for events.events.k8s.io
I0211 11:22:34.557687 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0211 11:22:34.557758 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io
I0211 11:22:34.557846 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for statefulsets.apps
I0211 11:22:34.557874 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for podtemplates
I0211 11:22:34.557891 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for limitranges
I0211 11:22:34.557914 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
I0211 11:22:34.557939 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for csistoragecapacities.storage.k8s.io
I0211 11:22:34.557958 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io
I0211 11:22:34.557976 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for addons.k3s.cattle.io
I0211 11:22:34.557993 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpoints
I0211 11:22:34.558008 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for daemonsets.apps
I0211 11:22:34.558026 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0211 11:22:34.558048 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for cronjobs.batch
I0211 11:22:34.558067 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
I0211 11:22:34.558087 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
I0211 11:22:34.558128 7 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0211 11:22:34.558142 7 controllermanager.go:577] Started "resourcequota"
W0211 11:22:34.558149 7 controllermanager.go:556] "tokencleaner" is disabled
I0211 11:22:34.558153 7 resource_quota_controller.go:273] Starting resource quota controller
I0211 11:22:34.558159 7 shared_informer.go:240] Waiting for caches to sync for resource quota
I0211 11:22:34.558175 7 resource_quota_monitor.go:304] QuotaMonitor running
I0211 11:22:34.699539 7 controllermanager.go:577] Started "persistentvolume-binder"
I0211 11:22:34.699566 7 pv_controller_base.go:308] Starting persistent volume controller
I0211 11:22:34.699569 7 shared_informer.go:240] Waiting for caches to sync for persistent volume
I0211 11:22:34.850699 7 controllermanager.go:577] Started "ttl-after-finished"
I0211 11:22:34.850746 7 ttlafterfinished_controller.go:109] Starting TTL after finished controller
I0211 11:22:34.850753 7 shared_informer.go:240] Waiting for caches to sync for TTL after finished
time="2022-02-11T11:22:34Z" level=info msg="Starting /v1, Kind=Secret controller"
I0211 11:22:35.048778 7 controllermanager.go:577] Started "disruption"
I0211 11:22:35.048795 7 disruption.go:363] Starting disruption controller
I0211 11:22:35.048805 7 shared_informer.go:240] Waiting for caches to sync for disruption
time="2022-02-11T11:22:35Z" level=info msg="Active TLS secret k3s-serving (ver=245) (count 10): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.29.0.3:172.29.0.3 listener.cattle.io/cn-k3d-foo-server-0:k3d-foo-server-0 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=FB473CA2A839175D300220BA26E34D832FC636FF]"
time="2022-02-11T11:22:35Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
I0211 11:22:35.199823 7 node_ipam_controller.go:91] Sending events to api server.
time="2022-02-11T11:22:35Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
I0211 11:22:35.994504 7 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
time="2022-02-11T11:22:35Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"ecfcacaa-fa8a-46a4-b970-263da2515a01\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"258\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"ecfcacaa-fa8a-46a4-b970-263da2515a01\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"258\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"bc03394a-223c-4bc9-9e86-ef1295d8e913\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"266\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
I0211 11:22:36.057938 7 controller.go:611] quota admission added evaluator for: deployments.apps
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"bc03394a-223c-4bc9-9e86-ef1295d8e913\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"266\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"local-storage\", UID:\"84c00b68-3eab-41e9-8967-65e4c77af581\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"279\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"local-storage\", UID:\"84c00b68-3eab-41e9-8967-65e4c77af581\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"279\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"aggregated-metrics-reader\", UID:\"d5a02827-d562-42d7-95ca-918fc4d4538e\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"291\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"aggregated-metrics-reader\", UID:\"d5a02827-d562-42d7-95ca-918fc4d4538e\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"291\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-delegator\", UID:\"20435d77-8fcf-4f3f-857b-7f96650d6a62\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"296\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-delegator\", UID:\"20435d77-8fcf-4f3f-857b-7f96650d6a62\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"296\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-reader\", UID:\"aec48a09-b24d-44ff-97f0-7305278b3019\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"301\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-reader\", UID:\"aec48a09-b24d-44ff-97f0-7305278b3019\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"301\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-apiservice\", UID:\"62cde126-90f9-4c07-bf14-84aa6a1c4517\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"306\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-apiservice\", UID:\"62cde126-90f9-4c07-bf14-84aa6a1c4517\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"306\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\""
I0211 11:22:36.570885 7 serving.go:354] Generated self-signed cert in-memory
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-deployment\", UID:\"442a2218-08e2-4808-879e-ddf40811aec7\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"312\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\""
time="2022-02-11T11:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-deployment\", UID:\"442a2218-08e2-4808-879e-ddf40811aec7\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"312\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\""
I0211 11:22:36.847736 7 controllermanager.go:142] Version: v1.22.2+k3s2
I0211 11:22:36.850074 7 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0211 11:22:36.850078 7 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0211 11:22:36.850087 7 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0211 11:22:36.850088 7 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0211 11:22:36.850078 7 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0211 11:22:36.850101 7 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0211 11:22:36.850397 7 secure_serving.go:200] Serving securely on 127.0.0.1:10258
I0211 11:22:36.850409 7 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0211 11:22:36.950961 7 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0211 11:22:36.950968 7 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0211 11:22:36.950968 7 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
time="2022-02-11T11:22:37Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-service\", UID:\"5db84949-85e8-4b54-874b-ea2af2beaf00\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"320\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\""
time="2022-02-11T11:22:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-service\", UID:\"5db84949-85e8-4b54-874b-ea2af2beaf00\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"320\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\""
time="2022-02-11T11:22:37Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"resource-reader\", UID:\"5c7d7b3a-32aa-4c82-9fe2-6e048d2e284b\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"327\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\""
time="2022-02-11T11:22:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"resource-reader\", UID:\"5c7d7b3a-32aa-4c82-9fe2-6e048d2e284b\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"327\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\""
W0211 11:22:37.832437 7 handler_proxy.go:104] no RequestInfo found in the context
E0211 11:22:37.832471 7 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0211 11:22:37.832479 7 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
time="2022-02-11T11:22:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"c88ae796-b809-40c3-bfe4-5b11503d12c8\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"333\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\""
time="2022-02-11T11:22:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"c88ae796-b809-40c3-bfe4-5b11503d12c8\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"333\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\""
time="2022-02-11T11:22:38Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
I0211 11:22:38.156014 7 request.go:665] Waited for 1.048512208s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6444/apis/k3s.cattle.io/v1?timeout=32s
time="2022-02-11T11:22:38Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"f7ba0309-4848-49e0-95e8-c931fab84dc7\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"340\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\""
I0211 11:22:38.430224 7 controller.go:611] quota admission added evaluator for: helmcharts.helm.cattle.io
time="2022-02-11T11:22:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"f7ba0309-4848-49e0-95e8-c931fab84dc7\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"340\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\""
I0211 11:22:38.458578 7 controller.go:611] quota admission added evaluator for: jobs.batch
E0211 11:22:38.657690 7 controllermanager.go:419] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0211 11:22:38.658948 7 node_controller.go:115] Sending events to api server.
I0211 11:22:38.659027 7 controllermanager.go:285] Started "cloud-node"
I0211 11:22:38.659096 7 node_controller.go:154] Waiting for informer caches to sync
I0211 11:22:38.660047 7 node_lifecycle_controller.go:76] Sending events to api server
I0211 11:22:38.660071 7 controllermanager.go:285] Started "cloud-node-lifecycle"
time="2022-02-11T11:22:39Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:39Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:40Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:40Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:41Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:41Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:42Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:42Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:43Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:43Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
time="2022-02-11T11:22:44Z" level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-02-11T11:22:44Z" level=info msg="Waiting for control-plane node k3d-foo-server-0 startup: nodes \"k3d-foo-server-0\" not found"
To me, this suggests a version mismatch between the kubelet and the containerd, but that seems weird since I'd assume that k3d packages both of those together.
Which OS & Architecture
- Arch Linux, amd64
This problem seems to have started after I did a system upgrade (pacman -Syu) at 2022-02-09, however no Docker-related packages (docker, runc, containerd) have been updated since way before then.
Which version of k3d
- k3d version v5.2.2
- k3s version v1.22.2-k3s2 (default)
Installed from Nixpkgs, I have also confirmed against k3d 5.3.0 and 5.2.1
Which version of docker
$ docker version
Client:
Version: 20.10.12
API version: 1.41
Go version: go1.17.5
Git commit: e91ed5707e
Built: Mon Dec 13 22:31:40 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.17.5
Git commit: 459d0dfbbb
Built: Mon Dec 13 22:30:43 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.5.9
GitCommit: 1407cab509ff0d96baa4f0eb6ff9980270e6e620.m
runc:
Version: 1.1.0
GitCommit: v1.1.0-0-g067aaf85
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
compose: Docker Compose (Docker Inc., 2.2.3)
Server:
Containers: 8
Running: 8
Paused: 0
Stopped: 0
Images: 91
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1407cab509ff0d96baa4f0eb6ff9980270e6e620.m
runc version: v1.1.0-0-g067aaf85
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.16.7-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 62.78GiB
Name: teo-dator-newarch
ID: SU7A:QWB6:6URN:WGBD:PLAH:N33I:2POH:2LBF:FK6R:64TF:JHJQ:EAFT
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: teozkr
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Stopping my old k3d cluster that was running in the background seems to have resolved the issue, and new clusters create fine again.
Curiously, kind also failed to create any clusters, with the following node logs:
docker logs kind-control-plane -f
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.23.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
systemd 248.3-1ubuntu8.1 running in system mode. (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Ubuntu 21.10!
Failed to create control group inotify object: Too many open files
Failed to allocate manager object: Too many open files
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.23.0.2
INFO: Detected old IPv4 address: 172.23.0.2
sed: can't read /etc/kubernetes/manifests/*.yaml: No such file or directory
sed: can't read /var/lib/kubelet/kubeadm-flags.env: No such file or directory
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
INFO: Detected old IPv6 address: fc00:f853:ccd:e793::2
sed: can't read /etc/kubernetes/manifests/*.yaml: No such file or directory
sed: can't read /var/lib/kubelet/kubeadm-flags.env: No such file or directory
systemd 248.3-1ubuntu8.1 running in system mode. (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Ubuntu 21.10!
Failed to create control group inotify object: Too many open files
Failed to allocate manager object: Too many open files
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
This seems to suggest that the k3d cluster was leaking inotify watches somewhere.. My inotify limits are already quite astronomical:
$ sudo sysctl -a | grep inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 524288
user.max_inotify_instances = 1024
user.max_inotify_watches = 524288
Hi @teozkr , thanks for opening this issue! Could it be, that your system update did the switch from cgroups v1 to v2?
To me, this suggests a version mismatch between the kubelet and the containerd, but that seems weird since I'd assume that k3d packages both of those together.
Packaging is not k3d but K3s (htttps://github.com/k3s-io/k3s), but nope, that likely won't happen there. Edit: there are some issues mentioning that error related to the underlying filesystem, but this doesn't seem to match here (see https://github.com/k3s-io/k3s/issues?q=is%3Aissue+sort%3Aupdated-desc+runtime.v1alpha2.RuntimeService+is%3Aclosed)
Stopping my old k3d cluster that was running in the background seems to have resolved the issue, and new clusters create fine again.
So the error occurred with a cluster that you created before the update and does not occur with new clusters you created after the update? (that would fit the cgroups theory)
Curiously, kind also failed to create any clusters, This seems to suggest that the k3d cluster was leaking inotify watches somewhere.
Not sure I can follow here. For sure that dangling k3d cluster was taking some resources, but I wouldn't think it was taking that much... but then again it was malfunctioning, so who knows :man_shrugging:
All in all... is there still an issue? :thinking:
So the error occurred with a cluster that you created before the update and does not occur with new clusters you created after the update? (that would fit the cgroups theory)
The error went away after (what was essentially) a k3d cluster stop && k3d cluster start, without clearing the cluster state with k3d cluster rm. I also rebooted after the system upgrade, which I'd assume would have had the same effect for clearing cgroups state.
All in all... is there still an issue? :thinking:
Hm, it seems to have returned now when trying to start a new cluster. My "main" cluster (1 server + 3 agents + 1 LB) has been running for ~39h, which seems to reinforce the resource leak hypothesis.
That sounds weird :thinking: I have multiple clusters continuously running with the following settings:
$ ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 254331
-n: file descriptors 1024
-l: locked-in-memory size (kbytes) 8156820
-v: address space (kbytes) unlimited
-x: file locks unlimited
-i: pending signals 254331
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0
-N 15: unlimited
$ sudo sysctl -a | grep inotify
Alias tip: _ sysctl -a | grep inotify
[sudo] password for thklein:
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 262144
user.max_inotify_instances = 128
user.max_inotify_watches = 262144
Would be cool, if you could figure out how many resources that cluster is really eating :thinking:
In a quick test, sudo lsof -a -p 55172 (PID of k3s container) revealed around 390 used filedescriptors (mostly socket connections).
$ ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256755
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 99
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256755
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I'll try lsof the next time I see the problem cropping up...
So your limits are even way higher :thinking: Would be really interesting to see more info on the actual resource usage there :+1:
lsof -a -p (k3s server)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
k3s 3476174 root cwd DIR 259,5 4096 7874620 /var/lib/rancher/k3s/server
k3s 3476174 root rtd DIR 0,88 4096 25690242 /
k3s 3476174 root txt REG 0,88 130823888 7491470 /bin/containerd
k3s 3476174 root mem REG 259,5 7491470 /bin/containerd (path inode=9490509)
k3s 3476174 root mem-r REG 259,5 7874638 /var/lib/rancher/k3s/server/db/state.db-shm (stat: No such file or directory)
k3s 3476174 root 0u CHR 1,3 0t0 197 /dev/null
k3s 3476174 root 1w FIFO 0,13 0t0 40815367 pipe
k3s 3476174 root 2w FIFO 0,13 0t0 40815368 pipe
k3s 3476174 root 3w CHR 1,3 0t0 197 /dev/null
k3s 3476174 root 4u a_inode 0,14 0 10579 [eventpoll:5,7,26,58,91,107,133,159,161,165,259,313,324,327,338,350,351,358,359,426,432,457,458,473,480,525,544,546,549,572,589,604...]
k3s 3476174 root 5r FIFO 0,13 0t0 40808213 pipe
k3s 3476174 root 6w FIFO 0,13 0t0 40808213 pipe
k3s 3476174 root 7u sock 0,8 0t0 40808217 protocol: UNIX
k3s 3476174 root 8u sock 0,8 0t0 40820371 protocol: TCPv6
k3s 3476174 root 9ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 10ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 11u REG 259,5 30277912 7874633 /var/lib/rancher/k3s/server/db/state.db-wal
k3s 3476174 root 12ur REG 259,5 32768 7874638 /var/lib/rancher/k3s/server/db/state.db-shm
k3s 3476174 root 13u sock 0,8 0t0 40813278 protocol: UNIX-STREAM
k3s 3476174 root 14u sock 0,8 0t0 40820374 protocol: UNIX-STREAM
k3s 3476174 root 15ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 16u sock 0,8 0t0 40805356 protocol: UNIX-STREAM
k3s 3476174 root 17ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 18u sock 0,8 0t0 40821232 protocol: UNIX-STREAM
k3s 3476174 root 19u sock 0,8 0t0 40820377 protocol: TCP
k3s 3476174 root 20u sock 0,8 0t0 40815392 protocol: UNIX-STREAM
k3s 3476174 root 21u sock 0,8 0t0 40820378 protocol: UNIX-STREAM
k3s 3476174 root 22u sock 0,8 0t0 40818075 protocol: UNIX-STREAM
k3s 3476174 root 23u sock 0,8 0t0 40822980 protocol: UNIX-STREAM
k3s 3476174 root 24u sock 0,8 0t0 40817213 protocol: UNIX-STREAM
k3s 3476174 root 25ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 26u sock 0,8 0t0 40809346 protocol: UNIX-STREAM
k3s 3476174 root 27ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 28ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 29u sock 0,8 0t0 40811261 protocol: UNIX-STREAM
k3s 3476174 root 30u sock 0,8 0t0 40804201 protocol: UNIX-STREAM
k3s 3476174 root 31u sock 0,8 0t0 40814471 protocol: UNIX-STREAM
k3s 3476174 root 32u sock 0,8 0t0 40805292 protocol: UNIX-STREAM
k3s 3476174 root 33u sock 0,8 0t0 40822181 protocol: UNIX-STREAM
k3s 3476174 root 34u sock 0,8 0t0 40817215 protocol: UNIX-STREAM
k3s 3476174 root 35u sock 0,8 0t0 40817217 protocol: UNIX-STREAM
k3s 3476174 root 36u sock 0,8 0t0 40824008 protocol: UNIX-STREAM
k3s 3476174 root 37ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 38ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 39u sock 0,8 0t0 40811240 protocol: UNIX-STREAM
k3s 3476174 root 40u sock 0,8 0t0 40822982 protocol: UNIX-STREAM
k3s 3476174 root 41u sock 0,8 0t0 40821233 protocol: UNIX-STREAM
k3s 3476174 root 42u sock 0,8 0t0 40822984 protocol: UNIX-STREAM
k3s 3476174 root 43u sock 0,8 0t0 40818079 protocol: UNIX-STREAM
k3s 3476174 root 44ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 45ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 46ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 47u sock 0,8 0t0 40825892 protocol: UNIX-STREAM
k3s 3476174 root 48u sock 0,8 0t0 40817218 protocol: UNIX-STREAM
k3s 3476174 root 49ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 50ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 51u sock 0,8 0t0 40817220 protocol: UNIX-STREAM
k3s 3476174 root 52u sock 0,8 0t0 40820379 protocol: UNIX-STREAM
k3s 3476174 root 53ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 54u sock 0,8 0t0 40824952 protocol: UNIX-STREAM
k3s 3476174 root 55u sock 0,8 0t0 40813275 protocol: UNIX-STREAM
k3s 3476174 root 56ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 57ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 58u sock 0,8 0t0 40809340 protocol: UNIX-STREAM
k3s 3476174 root 59u sock 0,8 0t0 40810318 protocol: UNIX-STREAM
k3s 3476174 root 60ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 61ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 62u sock 0,8 0t0 40804202 protocol: UNIX-STREAM
k3s 3476174 root 63u sock 0,8 0t0 40813277 protocol: UNIX-STREAM
k3s 3476174 root 64u sock 0,8 0t0 40819217 protocol: UNIX-STREAM
k3s 3476174 root 65u sock 0,8 0t0 40816291 protocol: TCP
k3s 3476174 root 66u sock 0,8 0t0 40819219 protocol: UNIX-STREAM
k3s 3476174 root 67u sock 0,8 0t0 40819235 protocol: TCP
k3s 3476174 root 68u sock 0,8 0t0 40806215 protocol: UNIX-STREAM
k3s 3476174 root 69ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 70ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 71ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 72u sock 0,8 0t0 40825893 protocol: UNIX-STREAM
k3s 3476174 root 73u sock 0,8 0t0 40815385 protocol: UNIX-STREAM
k3s 3476174 root 74u sock 0,8 0t0 40803064 protocol: UNIX-STREAM
k3s 3476174 root 75u sock 0,8 0t0 40815387 protocol: UNIX-STREAM
k3s 3476174 root 76u sock 0,8 0t0 40824010 protocol: UNIX-STREAM
k3s 3476174 root 77u sock 0,8 0t0 40815389 protocol: UNIX-STREAM
k3s 3476174 root 78u sock 0,8 0t0 40815391 protocol: UNIX-STREAM
k3s 3476174 root 79u sock 0,8 0t0 40824011 protocol: UNIX-STREAM
k3s 3476174 root 80ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 81u sock 0,8 0t0 40803065 protocol: UNIX-STREAM
k3s 3476174 root 82u sock 0,8 0t0 40824013 protocol: UNIX-STREAM
k3s 3476174 root 83ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 84u sock 0,8 0t0 40821229 protocol: UNIX-STREAM
k3s 3476174 root 85u sock 0,8 0t0 40803066 protocol: UNIX-STREAM
k3s 3476174 root 86ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 87u sock 0,8 0t0 40803068 protocol: UNIX-STREAM
k3s 3476174 root 88u sock 0,8 0t0 40821230 protocol: UNIX-STREAM
k3s 3476174 root 89ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 90u sock 0,8 0t0 40822985 protocol: UNIX-STREAM
k3s 3476174 root 91u sock 0,8 0t0 40819195 protocol: UNIX-STREAM
k3s 3476174 root 92u sock 0,8 0t0 40819220 protocol: UNIX-STREAM
k3s 3476174 root 93ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 94ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 95ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 96u sock 0,8 0t0 40812108 protocol: UNIX-STREAM
k3s 3476174 root 97ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 98u sock 0,8 0t0 40822182 protocol: UNIX-STREAM
k3s 3476174 root 99u sock 0,8 0t0 40814473 protocol: UNIX-STREAM
k3s 3476174 root 100u sock 0,8 0t0 40803089 protocol: UNIX-STREAM
k3s 3476174 root 101u sock 0,8 0t0 40805354 protocol: UNIX-STREAM
k3s 3476174 root 102u sock 0,8 0t0 40804203 protocol: UNIX-STREAM
k3s 3476174 root 103u sock 0,8 0t0 40805355 protocol: UNIX-STREAM
k3s 3476174 root 104u sock 0,8 0t0 40817221 protocol: UNIX-STREAM
k3s 3476174 root 105u sock 0,8 0t0 40819221 protocol: UNIX-STREAM
k3s 3476174 root 106u sock 0,8 0t0 40807311 protocol: UNIX-STREAM
k3s 3476174 root 107u sock 0,8 0t0 40824055 protocol: UNIX-STREAM
k3s 3476174 root 108u sock 0,8 0t0 40803091 protocol: UNIX-STREAM
k3s 3476174 root 109u sock 0,8 0t0 40821370 protocol: TCP
k3s 3476174 root 110u sock 0,8 0t0 40816285 protocol: UNIX-STREAM
k3s 3476174 root 111u sock 0,8 0t0 40803093 protocol: UNIX-STREAM
k3s 3476174 root 112u sock 0,8 0t0 40803095 protocol: UNIX-STREAM
k3s 3476174 root 113u sock 0,8 0t0 40804205 protocol: UNIX-STREAM
k3s 3476174 root 114u sock 0,8 0t0 40804207 protocol: UNIX-STREAM
k3s 3476174 root 115u sock 0,8 0t0 40822987 protocol: UNIX-STREAM
k3s 3476174 root 116u sock 0,8 0t0 40822989 protocol: UNIX-STREAM
k3s 3476174 root 117u sock 0,8 0t0 40812110 protocol: UNIX-STREAM
k3s 3476174 root 118u sock 0,8 0t0 40822994 protocol: TCPv6
k3s 3476174 root 119ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 120w REG 259,5 26735921 7874790 /var/lib/rancher/k3s/agent/containerd/containerd.log
k3s 3476174 root 121ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 122u sock 0,8 0t0 40821235 protocol: UNIX-STREAM
k3s 3476174 root 123u sock 0,8 0t0 40810320 protocol: UNIX-STREAM
k3s 3476174 root 124ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 125u sock 0,8 0t0 40821236 protocol: UNIX-STREAM
k3s 3476174 root 126u sock 0,8 0t0 40820381 protocol: UNIX-STREAM
k3s 3476174 root 127ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 128r FIFO 0,13 0t0 40805358 pipe
k3s 3476174 root 129u sock 0,8 0t0 40821244 protocol: UNIX-STREAM
k3s 3476174 root 130ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 131u sock 0,8 0t0 40830045 protocol: TCP
k3s 3476174 root 132u sock 0,8 0t0 40824972 protocol: UNIX-STREAM
k3s 3476174 root 133u sock 0,8 0t0 40811273 protocol: TCP
k3s 3476174 root 134u sock 0,8 0t0 40803136 protocol: TCPv6
k3s 3476174 root 135u sock 0,8 0t0 40832189 protocol: TCPv6
k3s 3476174 root 136u sock 0,8 0t0 40812497 protocol: TCP
k3s 3476174 root 137u sock 0,8 0t0 40824066 protocol: TCP
k3s 3476174 root 138u sock 0,8 0t0 199448024 protocol: TCPv6
k3s 3476174 root 139u sock 0,8 0t0 40818108 protocol: TCP
k3s 3476174 root 140u REG 259,5 30277912 7874633 /var/lib/rancher/k3s/server/db/state.db-wal
k3s 3476174 root 141ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 142ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 143u sock 0,8 0t0 40818123 protocol: TCP
k3s 3476174 root 144u sock 0,8 0t0 40825894 protocol: UNIX-STREAM
k3s 3476174 root 145ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 146u sock 0,8 0t0 40807350 protocol: TCP
k3s 3476174 root 147u sock 0,8 0t0 40804248 protocol: TCPv6
k3s 3476174 root 148u sock 0,8 0t0 40804250 protocol: TCP
k3s 3476174 root 149u sock 0,8 0t0 40824992 protocol: TCP
k3s 3476174 root 150ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 151u sock 0,8 0t0 40813307 protocol: TCPv6
k3s 3476174 root 152u sock 0,8 0t0 40811280 protocol: TCPv6
k3s 3476174 root 153ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 154u sock 0,8 0t0 40808245 protocol: TCP
k3s 3476174 root 155u sock 0,8 0t0 40819222 protocol: UNIX-STREAM
k3s 3476174 root 156u sock 0,8 0t0 40803096 protocol: UNIX-STREAM
k3s 3476174 root 157u sock 0,8 0t0 40804208 protocol: UNIX-STREAM
k3s 3476174 root 158u sock 0,8 0t0 40822990 protocol: UNIX-STREAM
k3s 3476174 root 159u sock 0,8 0t0 40809341 protocol: UNIX-STREAM
k3s 3476174 root 160u sock 0,8 0t0 40825895 protocol: UNIX-STREAM
k3s 3476174 root 161u sock 0,8 0t0 40809342 protocol: UNIX-STREAM
k3s 3476174 root 162u sock 0,8 0t0 40807313 protocol: UNIX-STREAM
k3s 3476174 root 163ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 164ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 165u sock 0,8 0t0 40809343 protocol: UNIX-STREAM
k3s 3476174 root 166u sock 0,8 0t0 40806217 protocol: UNIX-STREAM
k3s 3476174 root 167u sock 0,8 0t0 40808226 protocol: UNIX-STREAM
k3s 3476174 root 168u sock 0,8 0t0 40807315 protocol: UNIX-STREAM
k3s 3476174 root 169ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 170ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 171ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 172u sock 0,8 0t0 40824953 protocol: UNIX-STREAM
k3s 3476174 root 173u sock 0,8 0t0 40825897 protocol: UNIX-STREAM
k3s 3476174 root 174ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 175ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 176ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 177u sock 0,8 0t0 40807317 protocol: UNIX-STREAM
k3s 3476174 root 178u sock 0,8 0t0 40811263 protocol: UNIX-STREAM
k3s 3476174 root 179u sock 0,8 0t0 40807318 protocol: UNIX-STREAM
k3s 3476174 root 180u sock 0,8 0t0 40810322 protocol: UNIX-STREAM
k3s 3476174 root 181ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 182ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 183u sock 0,8 0t0 40807319 protocol: UNIX-STREAM
k3s 3476174 root 184u sock 0,8 0t0 40820383 protocol: UNIX-STREAM
k3s 3476174 root 185ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 186ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 187u sock 0,8 0t0 40807320 protocol: UNIX-STREAM
k3s 3476174 root 188u sock 0,8 0t0 40807321 protocol: UNIX-STREAM
k3s 3476174 root 189ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 190u sock 0,8 0t0 40807323 protocol: UNIX-STREAM
k3s 3476174 root 191u sock 0,8 0t0 40807324 protocol: UNIX-STREAM
k3s 3476174 root 192ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 193ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 194u sock 0,8 0t0 40819224 protocol: UNIX-STREAM
k3s 3476174 root 195u sock 0,8 0t0 40803098 protocol: UNIX-STREAM
k3s 3476174 root 196ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 197ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 198ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 199ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 200ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 201ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 202u sock 0,8 0t0 40803100 protocol: UNIX-STREAM
k3s 3476174 root 203u sock 0,8 0t0 40819225 protocol: UNIX-STREAM
k3s 3476174 root 204ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 205ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 206ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 207ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 208ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 209u sock 0,8 0t0 40819227 protocol: UNIX-STREAM
k3s 3476174 root 210u sock 0,8 0t0 40819228 protocol: UNIX-STREAM
k3s 3476174 root 211u sock 0,8 0t0 40819230 protocol: UNIX-STREAM
k3s 3476174 root 212u sock 0,8 0t0 40819231 protocol: UNIX-STREAM
k3s 3476174 root 213u sock 0,8 0t0 40819233 protocol: UNIX-STREAM
k3s 3476174 root 214u sock 0,8 0t0 40807326 protocol: UNIX-STREAM
k3s 3476174 root 215u sock 0,8 0t0 40807328 protocol: UNIX-STREAM
k3s 3476174 root 216u sock 0,8 0t0 40822183 protocol: UNIX-STREAM
k3s 3476174 root 217ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 218ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 219u sock 0,8 0t0 40807329 protocol: UNIX-STREAM
k3s 3476174 root 220u sock 0,8 0t0 40811265 protocol: UNIX-STREAM
k3s 3476174 root 221ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 222u sock 0,8 0t0 40807330 protocol: UNIX-STREAM
k3s 3476174 root 223u sock 0,8 0t0 40808227 protocol: UNIX-STREAM
k3s 3476174 root 224ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 225u sock 0,8 0t0 40807331 protocol: UNIX-STREAM
k3s 3476174 root 226u sock 0,8 0t0 40824954 protocol: UNIX-STREAM
k3s 3476174 root 227ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 228u sock 0,8 0t0 40824956 protocol: UNIX-STREAM
k3s 3476174 root 229u sock 0,8 0t0 40803101 protocol: UNIX-STREAM
k3s 3476174 root 230u sock 0,8 0t0 40826894 protocol: TCPv6
k3s 3476174 root 231ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 232u sock 0,8 0t0 40803103 protocol: UNIX-STREAM
k3s 3476174 root 233ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 234u sock 0,8 0t0 40803104 protocol: UNIX-STREAM
k3s 3476174 root 235ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 236u sock 0,8 0t0 40806258 protocol: TCP
k3s 3476174 root 237u sock 0,8 0t0 40803106 protocol: UNIX-STREAM
k3s 3476174 root 238u sock 0,8 0t0 40803107 protocol: UNIX-STREAM
k3s 3476174 root 239u sock 0,8 0t0 40803109 protocol: UNIX-STREAM
k3s 3476174 root 240u sock 0,8 0t0 40816286 protocol: UNIX-STREAM
k3s 3476174 root 241ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 242u sock 0,8 0t0 40810399 protocol: TCPv6
k3s 3476174 root 243ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 244u sock 0,8 0t0 40810402 protocol: TCP
k3s 3476174 root 245u sock 0,8 0t0 40806259 protocol: TCP
k3s 3476174 root 246u sock 0,8 0t0 40828980 protocol: UNIX-STREAM
k3s 3476174 root 247u sock 0,8 0t0 40817474 protocol: UNIX-STREAM
k3s 3476174 root 248u sock 0,8 0t0 40808388 protocol: UNIX-STREAM
k3s 3476174 root 249ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 250ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 251u sock 0,8 0t0 40810403 protocol: UNIX-STREAM
k3s 3476174 root 252u sock 0,8 0t0 40816288 protocol: UNIX-STREAM
k3s 3476174 root 253ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 254ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 255u sock 0,8 0t0 40816289 protocol: UNIX-STREAM
k3s 3476174 root 256u sock 0,8 0t0 40816290 protocol: UNIX-STREAM
k3s 3476174 root 257u sock 0,8 0t0 40832079 protocol: TCP
k3s 3476174 root 258u sock 0,8 0t0 40820575 protocol: TCP
k3s 3476174 root 259u sock 0,8 0t0 40810413 protocol: TCP
k3s 3476174 root 260u sock 0,8 0t0 40819554 protocol: TCP
k3s 3476174 root 261u sock 0,8 0t0 40812377 protocol: TCP
k3s 3476174 root 262u sock 0,8 0t0 40833057 protocol: TCP
k3s 3476174 root 263u sock 0,8 0t0 40831037 protocol: TCP
k3s 3476174 root 264u sock 0,8 0t0 40828075 protocol: TCP
k3s 3476174 root 265ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 266u sock 0,8 0t0 40817519 protocol: TCP
k3s 3476174 root 267u sock 0,8 0t0 40806326 protocol: TCP
k3s 3476174 root 268u sock 0,8 0t0 40819556 protocol: UNIX-STREAM
k3s 3476174 root 269u sock 0,8 0t0 40806364 protocol: TCPv6
k3s 3476174 root 270ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 271u sock 0,8 0t0 40813500 protocol: TCP
k3s 3476174 root 272u sock 0,8 0t0 40806313 protocol: UNIX-STREAM
k3s 3476174 root 273u sock 0,8 0t0 40827308 protocol: TCPv6
k3s 3476174 root 274u sock 0,8 0t0 40826338 protocol: TCPv6
k3s 3476174 root 275u sock 0,8 0t0 40834049 protocol: TCPv6
k3s 3476174 root 276u sock 0,8 0t0 40837178 protocol: TCPv6
k3s 3476174 root 277u sock 0,8 0t0 40834051 protocol: TCPv6
k3s 3476174 root 278ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 279ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 280u sock 0,8 0t0 40826340 protocol: TCP
k3s 3476174 root 281ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 282u sock 0,8 0t0 40832188 protocol: TCP
k3s 3476174 root 283u sock 0,8 0t0 118119264 protocol: TCP
k3s 3476174 root 285u sock 0,8 0t0 40812112 protocol: UNIX-STREAM
k3s 3476174 root 286ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 287u sock 0,8 0t0 40832203 protocol: TCP
k3s 3476174 root 288ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 289u sock 0,8 0t0 40834144 protocol: TCP
k3s 3476174 root 290r a_inode 0,14 0 10579 inotify
k3s 3476174 root 291u sock 0,8 0t0 40825342 protocol: TCP
k3s 3476174 root 292u sock 0,8 0t0 40823393 protocol: TCP
k3s 3476174 root 293u sock 0,8 0t0 40825343 protocol: TCPv6
k3s 3476174 root 294u sock 0,8 0t0 40834146 protocol: TCP
k3s 3476174 root 295u DIR 0,28 0 153964 /sys/fs/cgroup
k3s 3476174 root 296ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 297ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 298u sock 0,8 0t0 40832208 protocol: UNIX-STREAM
k3s 3476174 root 299r CHR 1,11 0t0 187 /dev/kmsg
k3s 3476174 root 300u sock 0,8 0t0 40808229 protocol: UNIX-STREAM
k3s 3476174 root 301u sock 0,8 0t0 40808230 protocol: UNIX-STREAM
k3s 3476174 root 302u sock 0,8 0t0 40808232 protocol: UNIX-STREAM
k3s 3476174 root 303u sock 0,8 0t0 40808233 protocol: UNIX-STREAM
k3s 3476174 root 304u sock 0,8 0t0 40811413 protocol: UNIX-STREAM
k3s 3476174 root 305u sock 0,8 0t0 40821237 protocol: UNIX-STREAM
k3s 3476174 root 306u sock 0,8 0t0 40814474 protocol: UNIX-STREAM
k3s 3476174 root 307u sock 0,8 0t0 40821238 protocol: UNIX-STREAM
k3s 3476174 root 308u sock 0,8 0t0 40814476 protocol: UNIX-STREAM
k3s 3476174 root 309u sock 0,8 0t0 40810324 protocol: UNIX-STREAM
k3s 3476174 root 310u sock 0,8 0t0 40807332 protocol: UNIX-STREAM
k3s 3476174 root 311u sock 0,8 0t0 40811267 protocol: UNIX-STREAM
k3s 3476174 root 312u sock 0,8 0t0 40811269 protocol: UNIX-STREAM
k3s 3476174 root 313u sock 0,8 0t0 40809344 protocol: UNIX-STREAM
k3s 3476174 root 314ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 315ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 316ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 317u sock 0,8 0t0 40822185 protocol: UNIX-STREAM
k3s 3476174 root 318ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 319u sock 0,8 0t0 40818081 protocol: UNIX-STREAM
k3s 3476174 root 320ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 321u sock 0,8 0t0 40813280 protocol: UNIX-STREAM
k3s 3476174 root 322u sock 0,8 0t0 40808235 protocol: UNIX-STREAM
k3s 3476174 root 323ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 324u sock 0,8 0t0 40824056 protocol: UNIX-STREAM
k3s 3476174 root 325ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 326u sock 0,8 0t0 40803110 protocol: UNIX-STREAM
k3s 3476174 root 327u sock 0,8 0t0 40831148 protocol: TCPv6
k3s 3476174 root 328ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 329ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 330u sock 0,8 0t0 40807333 protocol: UNIX-STREAM
k3s 3476174 root 331u sock 0,8 0t0 40813281 protocol: UNIX-STREAM
k3s 3476174 root 332ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 333u a_inode 0,14 0 10579 [eventpoll:335,336]
k3s 3476174 root 334ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 335r a_inode 0,14 0 10579 inotify
k3s 3476174 root 336r FIFO 0,13 0t0 40825346 pipe
k3s 3476174 root 337w FIFO 0,13 0t0 40825346 pipe
k3s 3476174 root 338u sock 0,8 0t0 40831151 protocol: TCP
k3s 3476174 root 339ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 340u sock 0,8 0t0 40827139 protocol: TCPv6
k3s 3476174 root 341u sock 0,8 0t0 40828044 protocol: TCP
k3s 3476174 root 342ur REG 259,5 113102848 7874632 /var/lib/rancher/k3s/server/db/state.db
k3s 3476174 root 343u sock 0,8 0t0 40823379 protocol: TCPv6
k3s 3476174 root 344r a_inode 0,14 0 10579 inotify
k3s 3476174 root 345u a_inode 0,14 0 10579 [eventpoll:344,346]
k3s 3476174 root 346r FIFO 0,13 0t0 40837139 pipe
k3s 3476174 root 347w FIFO 0,13 0t0 40837139 pipe
k3s 3476174 root 348u sock 0,8 0t0 40835146 protocol: TCP
k3s 3476174 root 350u sock 0,8 0t0 199116010 protocol: TCPv6
k3s 3476174 root 351u sock 0,8 0t0 196384241 protocol: TCPv6
k3s 3476174 root 352u sock 0,8 0t0 117585338 protocol: TCPv6
k3s 3476174 root 353u sock 0,8 0t0 40821486 protocol: TCPv6
k3s 3476174 root 354u sock 0,8 0t0 40827141 protocol: UNIX-STREAM
k3s 3476174 root 355u sock 0,8 0t0 40835149 protocol: TCPv6
k3s 3476174 root 356u sock 0,8 0t0 40826341 protocol: TCP
k3s 3476174 root 357r a_inode 0,14 0 10579 inotify
k3s 3476174 root 358u sock 0,8 0t0 40837142 protocol: UNIX-STREAM
k3s 3476174 root 359u sock 0,8 0t0 40837041 protocol: TCPv6
k3s 3476174 root 360u sock 0,8 0t0 40824252 protocol: TCP
k3s 3476174 root 361u sock 0,8 0t0 40821485 protocol: TCP
k3s 3476174 root 362u sock 0,8 0t0 40835151 protocol: TCPv6
k3s 3476174 root 363u sock 0,8 0t0 40822685 protocol: TCPv6
k3s 3476174 root 364u sock 0,8 0t0 40828076 protocol: TCP
k3s 3476174 root 365u sock 0,8 0t0 40829047 protocol: UNIX-STREAM
k3s 3476174 root 366r a_inode 0,14 0 10579 inotify
k3s 3476174 root 367r CHR 1,11 0t0 187 /dev/kmsg
k3s 3476174 root 368u sock 0,8 0t0 40828077 protocol: TCP
k3s 3476174 root 369u sock 0,8 0t0 40828078 protocol: TCP
k3s 3476174 root 370u sock 0,8 0t0 40828079 protocol: TCP
k3s 3476174 root 371r FIFO 0,13 0t0 40829048 pipe
k3s 3476174 root 372u a_inode 0,14 0 10579 [eventpoll:290,371]
k3s 3476174 root 373w FIFO 0,13 0t0 40829048 pipe
k3s 3476174 root 374u sock 0,8 0t0 40828080 protocol: TCP
k3s 3476174 root 375u sock 0,8 0t0 40828081 protocol: TCP
k3s 3476174 root 376u sock 0,8 0t0 40828082 protocol: TCP
k3s 3476174 root 377u sock 0,8 0t0 40828083 protocol: TCP
k3s 3476174 root 378u sock 0,8 0t0 40828084 protocol: TCP
k3s 3476174 root 379u sock 0,8 0t0 40828085 protocol: TCP
k3s 3476174 root 380u sock 0,8 0t0 40828086 protocol: TCP
k3s 3476174 root 381u sock 0,8 0t0 40828087 protocol: TCP
k3s 3476174 root 382u sock 0,8 0t0 40828088 protocol: TCP
k3s 3476174 root 383u sock 0,8 0t0 40828089 protocol: TCP
k3s 3476174 root 384u sock 0,8 0t0 40828090 protocol: TCP
k3s 3476174 root 385u sock 0,8 0t0 40828091 protocol: TCP
k3s 3476174 root 386u sock 0,8 0t0 40828092 protocol: TCP
k3s 3476174 root 387u sock 0,8 0t0 40828093 protocol: TCP
k3s 3476174 root 388u sock 0,8 0t0 40828094 protocol: TCP
k3s 3476174 root 389u sock 0,8 0t0 40828095 protocol: TCP
k3s 3476174 root 390u sock 0,8 0t0 40828096 protocol: TCP
k3s 3476174 root 391u sock 0,8 0t0 40828097 protocol: TCP
k3s 3476174 root 392u sock 0,8 0t0 40828098 protocol: TCP
k3s 3476174 root 393u sock 0,8 0t0 40828099 protocol: TCP
k3s 3476174 root 394u sock 0,8 0t0 40828100 protocol: TCP
k3s 3476174 root 395u sock 0,8 0t0 40828101 protocol: TCP
k3s 3476174 root 396u sock 0,8 0t0 40828102 protocol: TCP
k3s 3476174 root 397u sock 0,8 0t0 40828103 protocol: TCP
k3s 3476174 root 398u sock 0,8 0t0 40828104 protocol: TCP
k3s 3476174 root 399u sock 0,8 0t0 40828105 protocol: TCP
k3s 3476174 root 400u sock 0,8 0t0 40828106 protocol: TCP
k3s 3476174 root 401u sock 0,8 0t0 40828107 protocol: TCP
k3s 3476174 root 402u sock 0,8 0t0 40828108 protocol: TCP
k3s 3476174 root 403u sock 0,8 0t0 40828109 protocol: TCP
k3s 3476174 root 404u sock 0,8 0t0 40836263 protocol: TCP
k3s 3476174 root 405u sock 0,8 0t0 40816621 protocol: TCPv6
k3s 3476174 root 406u sock 0,8 0t0 40819620 protocol: TCP
k3s 3476174 root 407u sock 0,8 0t0 40830077 protocol: TCPv6
k3s 3476174 root 408u sock 0,8 0t0 40823392 protocol: TCP
k3s 3476174 root 409u sock 0,8 0t0 40821547 protocol: TCP
k3s 3476174 root 410r a_inode 0,14 0 10579 inotify
k3s 3476174 root 411u sock 0,8 0t0 40824297 protocol: TCP
k3s 3476174 root 412u a_inode 0,14 0 10579 [eventpoll:410,413]
k3s 3476174 root 413r FIFO 0,13 0t0 40811427 pipe
k3s 3476174 root 414w FIFO 0,13 0t0 40811427 pipe
k3s 3476174 root 415u sock 0,8 0t0 40822688 protocol: TCP
k3s 3476174 root 416u sock 0,8 0t0 40837181 protocol: TCPv6
k3s 3476174 root 417u sock 0,8 0t0 40837180 protocol: TCPv6
k3s 3476174 root 418u sock 0,8 0t0 40826342 protocol: TCPv6
k3s 3476174 root 419u sock 0,8 0t0 40811433 protocol: TCP
k3s 3476174 root 420u sock 0,8 0t0 117570691 protocol: TCP
k3s 3476174 root 421u sock 0,8 0t0 40826344 protocol: TCP
k3s 3476174 root 422u sock 0,8 0t0 40836277 protocol: TCPv6
k3s 3476174 root 423u REG 259,5 30277912 7874633 /var/lib/rancher/k3s/server/db/state.db-wal
k3s 3476174 root 424u sock 0,8 0t0 40836279 protocol: TCPv6
k3s 3476174 root 426u sock 0,8 0t0 199457804 protocol: TCP
k3s 3476174 root 427u sock 0,8 0t0 40831202 protocol: TCPv6
k3s 3476174 root 428u sock 0,8 0t0 40831204 protocol: TCPv6
k3s 3476174 root 429u sock 0,8 0t0 40827281 protocol: TCPv6
k3s 3476174 root 430u sock 0,8 0t0 40827287 protocol: TCPv6
k3s 3476174 root 431u sock 0,8 0t0 40827288 protocol: TCPv6
k3s 3476174 root 432u sock 0,8 0t0 40840207 protocol: TCPv6
k3s 3476174 root 433u sock 0,8 0t0 40839284 protocol: TCPv6
k3s 3476174 root 434u sock 0,8 0t0 40832378 protocol: TCPv6
k3s 3476174 root 435u sock 0,8 0t0 40832380 protocol: TCPv6
k3s 3476174 root 436u sock 0,8 0t0 40819682 protocol: TCPv6
k3s 3476174 root 437u sock 0,8 0t0 196208039 protocol: TCPv6
k3s 3476174 root 438u sock 0,8 0t0 40833420 protocol: TCPv6
k3s 3476174 root 439u sock 0,8 0t0 40825679 protocol: TCPv6
k3s 3476174 root 440u sock 0,8 0t0 40836309 protocol: TCPv6
k3s 3476174 root 441u sock 0,8 0t0 189857523 protocol: TCPv6
k3s 3476174 root 442u sock 0,8 0t0 46387711 protocol: TCP
k3s 3476174 root 443u sock 0,8 0t0 199451870 protocol: TCPv6
k3s 3476174 root 444u sock 0,8 0t0 40840227 protocol: TCPv6
k3s 3476174 root 445u sock 0,8 0t0 40840228 protocol: TCPv6
k3s 3476174 root 446u sock 0,8 0t0 40821615 protocol: TCPv6
k3s 3476174 root 448u sock 0,8 0t0 40833428 protocol: TCPv6
k3s 3476174 root 449u sock 0,8 0t0 46317499 protocol: UNIX-STREAM
k3s 3476174 root 450u sock 0,8 0t0 46341237 protocol: UNIX-STREAM
k3s 3476174 root 451u sock 0,8 0t0 40837234 protocol: TCPv6
k3s 3476174 root 452u sock 0,8 0t0 199451098 protocol: TCPv6
k3s 3476174 root 453u sock 0,8 0t0 40831225 protocol: TCPv6
k3s 3476174 root 454u sock 0,8 0t0 40831226 protocol: TCPv6
k3s 3476174 root 455u sock 0,8 0t0 40841276 protocol: TCPv6
k3s 3476174 root 456u sock 0,8 0t0 40811503 protocol: TCPv6
k3s 3476174 root 457u sock 0,8 0t0 40824545 protocol: TCPv6
k3s 3476174 root 458u sock 0,8 0t0 40824547 protocol: TCPv6
k3s 3476174 root 459u sock 0,8 0t0 40850543 protocol: TCPv6
k3s 3476174 root 460u sock 0,8 0t0 40843459 protocol: TCPv6
k3s 3476174 root 461u sock 0,8 0t0 40830476 protocol: TCPv6
k3s 3476174 root 462u sock 0,8 0t0 40829797 protocol: TCPv6
k3s 3476174 root 463u sock 0,8 0t0 40834916 protocol: TCPv6
k3s 3476174 root 464u sock 0,8 0t0 46333705 protocol: TCP
k3s 3476174 root 465u sock 0,8 0t0 46333706 protocol: TCP
k3s 3476174 root 466u sock 0,8 0t0 40838091 protocol: TCPv6
k3s 3476174 root 467u sock 0,8 0t0 40838092 protocol: TCPv6
k3s 3476174 root 468u sock 0,8 0t0 40838095 protocol: TCPv6
k3s 3476174 root 469u sock 0,8 0t0 40844730 protocol: TCPv6
k3s 3476174 root 470u sock 0,8 0t0 40850511 protocol: TCPv6
k3s 3476174 root 471u sock 0,8 0t0 199448074 protocol: TCPv6
k3s 3476174 root 472u sock 0,8 0t0 40845789 protocol: TCPv6
k3s 3476174 root 473u sock 0,8 0t0 46391652 protocol: TCPv6
k3s 3476174 root 475u sock 0,8 0t0 40840862 protocol: TCPv6
k3s 3476174 root 476u sock 0,8 0t0 40840863 protocol: TCPv6
k3s 3476174 root 479u sock 0,8 0t0 40850532 protocol: TCPv6
k3s 3476174 root 480u sock 0,8 0t0 40851700 protocol: TCPv6
k3s 3476174 root 481u sock 0,8 0t0 118124934 protocol: TCP
k3s 3476174 root 482u sock 0,8 0t0 40869895 protocol: UNIX-STREAM
k3s 3476174 root 483u sock 0,8 0t0 40849595 protocol: TCPv6
k3s 3476174 root 484u sock 0,8 0t0 40848800 protocol: TCPv6
k3s 3476174 root 485u sock 0,8 0t0 40828859 protocol: TCPv6
k3s 3476174 root 486u sock 0,8 0t0 40830774 protocol: TCPv6
k3s 3476174 root 487u sock 0,8 0t0 40842668 protocol: TCPv6
k3s 3476174 root 488u sock 0,8 0t0 40841862 protocol: TCPv6
k3s 3476174 root 489u sock 0,8 0t0 40849609 protocol: TCPv6
k3s 3476174 root 490u sock 0,8 0t0 40837017 protocol: TCPv6
k3s 3476174 root 491u sock 0,8 0t0 40845910 protocol: TCPv6
k3s 3476174 root 492u sock 0,8 0t0 173728050 protocol: TCPv6
k3s 3476174 root 493u sock 0,8 0t0 40845912 protocol: TCPv6
k3s 3476174 root 494u sock 0,8 0t0 40830853 protocol: TCPv6
k3s 3476174 root 495u sock 0,8 0t0 40830854 protocol: TCPv6
k3s 3476174 root 496u sock 0,8 0t0 40830855 protocol: TCPv6
k3s 3476174 root 497u sock 0,8 0t0 40855576 protocol: TCPv6
k3s 3476174 root 498u sock 0,8 0t0 40855577 protocol: TCPv6
k3s 3476174 root 500u sock 0,8 0t0 40855579 protocol: TCPv6
k3s 3476174 root 501u sock 0,8 0t0 40855580 protocol: TCPv6
k3s 3476174 root 503u sock 0,8 0t0 40832875 protocol: TCPv6
k3s 3476174 root 504u sock 0,8 0t0 40832876 protocol: TCPv6
k3s 3476174 root 505u sock 0,8 0t0 40850617 protocol: TCPv6
k3s 3476174 root 506u sock 0,8 0t0 40850618 protocol: TCPv6
k3s 3476174 root 507u sock 0,8 0t0 40848851 protocol: TCPv6
k3s 3476174 root 508u sock 0,8 0t0 40848852 protocol: TCPv6
k3s 3476174 root 509u sock 0,8 0t0 40866972 protocol: UNIX-STREAM
k3s 3476174 root 510u sock 0,8 0t0 40847898 protocol: TCPv6
k3s 3476174 root 511u sock 0,8 0t0 40830884 protocol: TCPv6
k3s 3476174 root 512u sock 0,8 0t0 40858081 protocol: UNIX-STREAM
k3s 3476174 root 513u sock 0,8 0t0 40845951 protocol: TCPv6
k3s 3476174 root 515u sock 0,8 0t0 40844895 protocol: TCPv6
k3s 3476174 root 517u sock 0,8 0t0 40836085 protocol: TCPv6
k3s 3476174 root 518u sock 0,8 0t0 40855777 protocol: TCPv6
k3s 3476174 root 519u sock 0,8 0t0 193003280 protocol: TCP
k3s 3476174 root 520u sock 0,8 0t0 40855781 protocol: TCPv6
k3s 3476174 root 521u sock 0,8 0t0 40853085 protocol: TCPv6
k3s 3476174 root 523u sock 0,8 0t0 40852148 protocol: UNIX-STREAM
k3s 3476174 root 524u sock 0,8 0t0 40866963 protocol: UNIX-STREAM
k3s 3476174 root 525u sock 0,8 0t0 40845249 protocol: UNIX-STREAM
k3s 3476174 root 526u sock 0,8 0t0 40850137 protocol: UNIX-STREAM
k3s 3476174 root 527u sock 0,8 0t0 40851029 protocol: UNIX-STREAM
k3s 3476174 root 528u sock 0,8 0t0 40861113 protocol: UNIX-STREAM
k3s 3476174 root 529u sock 0,8 0t0 40849262 protocol: UNIX-STREAM
k3s 3476174 root 530u sock 0,8 0t0 40869893 protocol: UNIX-STREAM
k3s 3476174 root 531u sock 0,8 0t0 40855263 protocol: UNIX-STREAM
k3s 3476174 root 532u sock 0,8 0t0 40870962 protocol: UNIX-STREAM
k3s 3476174 root 533u sock 0,8 0t0 40856124 protocol: UNIX-STREAM
k3s 3476174 root 534u sock 0,8 0t0 40867954 protocol: UNIX-STREAM
k3s 3476174 root 535u sock 0,8 0t0 40855279 protocol: UNIX-STREAM
k3s 3476174 root 536u sock 0,8 0t0 40856125 protocol: UNIX-STREAM
k3s 3476174 root 537u sock 0,8 0t0 40858093 protocol: UNIX-STREAM
k3s 3476174 root 538u sock 0,8 0t0 40867014 protocol: UNIX-STREAM
k3s 3476174 root 539u sock 0,8 0t0 40858094 protocol: UNIX-STREAM
k3s 3476174 root 540u sock 0,8 0t0 40856126 protocol: UNIX-STREAM
k3s 3476174 root 541u sock 0,8 0t0 40870954 protocol: UNIX-STREAM
k3s 3476174 root 542u sock 0,8 0t0 40872979 protocol: UNIX-STREAM
k3s 3476174 root 543u sock 0,8 0t0 40869943 protocol: UNIX-STREAM
k3s 3476174 root 544u sock 0,8 0t0 40849325 protocol: UNIX-STREAM
k3s 3476174 root 545u sock 0,8 0t0 40860358 protocol: UNIX-STREAM
k3s 3476174 root 546u sock 0,8 0t0 40858110 protocol: UNIX-STREAM
k3s 3476174 root 547u sock 0,8 0t0 40867024 protocol: UNIX-STREAM
k3s 3476174 root 548u sock 0,8 0t0 40860360 protocol: UNIX-STREAM
k3s 3476174 root 549u sock 0,8 0t0 40867959 protocol: UNIX-STREAM
k3s 3476174 root 550u sock 0,8 0t0 40864994 protocol: UNIX-STREAM
k3s 3476174 root 551u sock 0,8 0t0 40860362 protocol: UNIX-STREAM
k3s 3476174 root 552u sock 0,8 0t0 40860302 protocol: UNIX-STREAM
k3s 3476174 root 553u sock 0,8 0t0 40849346 protocol: UNIX-STREAM
k3s 3476174 root 554u sock 0,8 0t0 40865074 protocol: UNIX-STREAM
k3s 3476174 root 555u sock 0,8 0t0 40861219 protocol: UNIX-STREAM
k3s 3476174 root 556u sock 0,8 0t0 40858134 protocol: UNIX-STREAM
k3s 3476174 root 557u sock 0,8 0t0 40862339 protocol: UNIX-STREAM
k3s 3476174 root 558u sock 0,8 0t0 40854161 protocol: UNIX-STREAM
k3s 3476174 root 559u sock 0,8 0t0 40856201 protocol: UNIX-STREAM
k3s 3476174 root 560u sock 0,8 0t0 40871090 protocol: UNIX-STREAM
k3s 3476174 root 561u sock 0,8 0t0 40857332 protocol: UNIX-STREAM
k3s 3476174 root 562u sock 0,8 0t0 40871092 protocol: UNIX-STREAM
k3s 3476174 root 563u sock 0,8 0t0 40882223 protocol: TCPv6
k3s 3476174 root 564u sock 0,8 0t0 40868101 protocol: UNIX-STREAM
k3s 3476174 root 565u sock 0,8 0t0 40870072 protocol: UNIX-STREAM
k3s 3476174 root 566u sock 0,8 0t0 40866677 protocol: TCPv6
k3s 3476174 root 567u sock 0,8 0t0 40869042 protocol: UNIX-STREAM
k3s 3476174 root 568u sock 0,8 0t0 40850305 protocol: UNIX-STREAM
k3s 3476174 root 569u sock 0,8 0t0 40871100 protocol: UNIX-STREAM
k3s 3476174 root 570u sock 0,8 0t0 40876488 protocol: TCPv6
k3s 3476174 root 571u sock 0,8 0t0 117544574 protocol: UNIX-STREAM
k3s 3476174 root 572u sock 0,8 0t0 197178440 protocol: TCPv6
k3s 3476174 root 573u sock 0,8 0t0 40864528 protocol: TCPv6
k3s 3476174 root 574u sock 0,8 0t0 40903993 protocol: TCPv6
k3s 3476174 root 576u sock 0,8 0t0 40881272 protocol: TCPv6
k3s 3476174 root 577u sock 0,8 0t0 193011313 protocol: TCPv6
k3s 3476174 root 579u sock 0,8 0t0 40871597 protocol: TCPv6
k3s 3476174 root 580u sock 0,8 0t0 40880201 protocol: TCPv6
k3s 3476174 root 581u sock 0,8 0t0 40880202 protocol: TCPv6
k3s 3476174 root 582u sock 0,8 0t0 40880203 protocol: TCPv6
k3s 3476174 root 583u sock 0,8 0t0 40880204 protocol: TCPv6
k3s 3476174 root 584u sock 0,8 0t0 40880205 protocol: TCPv6
k3s 3476174 root 585u sock 0,8 0t0 40880206 protocol: TCPv6
k3s 3476174 root 586u sock 0,8 0t0 40880207 protocol: TCPv6
k3s 3476174 root 587u sock 0,8 0t0 40880208 protocol: TCPv6
k3s 3476174 root 588u sock 0,8 0t0 40880209 protocol: TCPv6
k3s 3476174 root 589u sock 0,8 0t0 117561736 protocol: UNIX-STREAM
k3s 3476174 root 591u sock 0,8 0t0 40877423 protocol: TCPv6
k3s 3476174 root 592u sock 0,8 0t0 40869443 protocol: TCPv6
k3s 3476174 root 593u sock 0,8 0t0 40864531 protocol: TCPv6
k3s 3476174 root 594u sock 0,8 0t0 40877426 protocol: TCPv6
k3s 3476174 root 595u sock 0,8 0t0 40882236 protocol: TCPv6
k3s 3476174 root 597u sock 0,8 0t0 40864535 protocol: TCPv6
k3s 3476174 root 598u sock 0,8 0t0 40864536 protocol: TCPv6
k3s 3476174 root 599u sock 0,8 0t0 40864537 protocol: TCPv6
k3s 3476174 root 600u sock 0,8 0t0 40864538 protocol: TCPv6
k3s 3476174 root 601u sock 0,8 0t0 40864539 protocol: TCPv6
k3s 3476174 root 602u sock 0,8 0t0 40864540 protocol: TCPv6
k3s 3476174 root 604u sock 0,8 0t0 40864542 protocol: TCPv6
k3s 3476174 root 605u sock 0,8 0t0 40864543 protocol: TCPv6
k3s 3476174 root 606u sock 0,8 0t0 40866678 protocol: TCPv6
k3s 3476174 root 607u sock 0,8 0t0 40898019 protocol: TCPv6
k3s 3476174 root 608u sock 0,8 0t0 40866679 protocol: TCPv6
k3s 3476174 root 609u sock 0,8 0t0 40872501 protocol: TCPv6
k3s 3476174 root 610u sock 0,8 0t0 40872502 protocol: TCPv6
k3s 3476174 root 611u sock 0,8 0t0 40872503 protocol: TCPv6
k3s 3476174 root 612u sock 0,8 0t0 40872504 protocol: TCPv6
k3s 3476174 root 613u sock 0,8 0t0 40872505 protocol: TCPv6
k3s 3476174 root 614u sock 0,8 0t0 40873509 protocol: TCPv6
k3s 3476174 root 615u sock 0,8 0t0 40873511 protocol: TCPv6
k3s 3476174 root 616u sock 0,8 0t0 40872508 protocol: TCPv6
k3s 3476174 root 617u sock 0,8 0t0 40877740 protocol: TCPv6
k3s 3476174 root 618u sock 0,8 0t0 40877741 protocol: TCPv6
k3s 3476174 root 619u sock 0,8 0t0 40877742 protocol: TCPv6
k3s 3476174 root 620u sock 0,8 0t0 40877743 protocol: TCPv6
k3s 3476174 root 621u sock 0,8 0t0 40877744 protocol: TCPv6
k3s 3476174 root 622u sock 0,8 0t0 40877745 protocol: TCPv6
k3s 3476174 root 623u sock 0,8 0t0 40873513 protocol: TCPv6
k3s 3476174 root 624u sock 0,8 0t0 40861652 protocol: TCPv6
k3s 3476174 root 625u sock 0,8 0t0 118155875 protocol: TCP
k3s 3476174 root 626u sock 0,8 0t0 118097416 protocol: TCP
k3s 3476174 root 628u sock 0,8 0t0 40898020 protocol: TCPv6
k3s 3476174 root 629u sock 0,8 0t0 40898021 protocol: TCPv6
k3s 3476174 root 630u sock 0,8 0t0 40898022 protocol: TCPv6
k3s 3476174 root 631u sock 0,8 0t0 40898023 protocol: TCPv6
k3s 3476174 root 632u sock 0,8 0t0 40898024 protocol: TCPv6
k3s 3476174 root 633u sock 0,8 0t0 40898025 protocol: TCPv6
k3s 3476174 root 635u sock 0,8 0t0 40898026 protocol: TCPv6
k3s 3476174 root 636u sock 0,8 0t0 40893921 protocol: TCPv6
k3s 3476174 root 637u sock 0,8 0t0 40893922 protocol: TCPv6
k3s 3476174 root 638u sock 0,8 0t0 40893923 protocol: TCPv6
k3s 3476174 root 639u sock 0,8 0t0 40893924 protocol: TCPv6
k3s 3476174 root 640u sock 0,8 0t0 40890346 protocol: TCPv6
k3s 3476174 root 641u sock 0,8 0t0 40903063 protocol: TCPv6
k3s 3476174 root 642u sock 0,8 0t0 40903065 protocol: TCPv6
k3s 3476174 root 643u sock 0,8 0t0 40903067 protocol: TCPv6
k3s 3476174 root 645u sock 0,8 0t0 118185223 protocol: TCP
k3s 3476174 root 647u sock 0,8 0t0 118132467 protocol: TCP
k3s 3476174 root 648u sock 0,8 0t0 40954237 protocol: TCPv6
k3s 3476174 root 649u sock 0,8 0t0 40954239 protocol: TCPv6
k3s 3476174 root 650u sock 0,8 0t0 40945168 protocol: TCPv6
k3s 3476174 root 658u sock 0,8 0t0 118149552 protocol: TCPv6
k3s 3476174 root 662u sock 0,8 0t0 117620017 protocol: TCPv6
k3s 3476174 root 664u sock 0,8 0t0 117620018 protocol: TCPv6
k3s 3476174 root 665u sock 0,8 0t0 117620020 protocol: TCPv6
k3s 3476174 root 666u sock 0,8 0t0 117616987 protocol: TCPv6
k3s 3476174 root 667u sock 0,8 0t0 117620022 protocol: TCPv6
k3s 3476174 root 670u sock 0,8 0t0 117616992 protocol: TCPv6
k3s 3476174 root 671u sock 0,8 0t0 117634994 protocol: TCPv6
k3s 3476174 root 674u sock 0,8 0t0 117620025 protocol: TCPv6
k3s 3476174 root 675u sock 0,8 0t0 117633145 protocol: TCPv6
lsof -a -p (k3s agent 1)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
k3s 3477252 root cwd DIR 0,98 4096 25690654 /
k3s 3477252 root rtd DIR 0,98 4096 25690654 /
k3s 3477252 root txt REG 0,98 130823888 7491470 /bin/containerd
k3s 3477252 root mem REG 259,5 7491470 /bin/containerd (path inode=9490509)
k3s 3477252 root 0u CHR 1,3 0t0 197 /dev/null
k3s 3477252 root 1w FIFO 0,13 0t0 40811343 pipe
k3s 3477252 root 2w FIFO 0,13 0t0 40811344 pipe
k3s 3477252 root 3w CHR 1,3 0t0 197 /dev/null
k3s 3477252 root 4u a_inode 0,14 0 10579 [eventpoll:8,17,33,35,41,42,43,48,52,59,65,66,67,68,78,79,80,81,82,117,130,131,156,163,165,176,177,188,189,243,262,304...]
k3s 3477252 root 5r FIFO 0,13 0t0 40823228 pipe
k3s 3477252 root 6w FIFO 0,13 0t0 40823228 pipe
k3s 3477252 root 7u sock 0,8 0t0 40806319 protocol: UNIX
k3s 3477252 root 8u sock 0,8 0t0 40813527 protocol: TCP
k3s 3477252 root 9u sock 0,8 0t0 40821429 protocol: TCP
k3s 3477252 root 10u sock 0,8 0t0 40823271 protocol: TCP
k3s 3477252 root 11u sock 0,8 0t0 40832164 protocol: TCP
k3s 3477252 root 12r FIFO 0,13 0t0 40823272 pipe
k3s 3477252 root 13w FIFO 0,13 0t0 40823272 pipe
k3s 3477252 root 14r FIFO 0,13 0t0 40823273 pipe
k3s 3477252 root 15w FIFO 0,13 0t0 40823273 pipe
k3s 3477252 root 16w REG 259,5 40944379 23210989 /var/lib/rancher/k3s/agent/containerd/containerd.log
k3s 3477252 root 17r FIFO 0,13 0t0 40812513 pipe
k3s 3477252 root 18u sock 0,8 0t0 40835248 protocol: TCP
k3s 3477252 root 19u sock 0,8 0t0 40826399 protocol: TCP
k3s 3477252 root 20u sock 0,8 0t0 40817651 protocol: TCP
k3s 3477252 root 21r FIFO 0,13 0t0 40817652 pipe
k3s 3477252 root 22w FIFO 0,13 0t0 40817652 pipe
k3s 3477252 root 23r FIFO 0,13 0t0 40835249 pipe
k3s 3477252 root 24w FIFO 0,13 0t0 40835249 pipe
k3s 3477252 root 25u sock 0,8 0t0 40817653 protocol: TCP
k3s 3477252 root 26r FIFO 0,13 0t0 40826407 pipe
k3s 3477252 root 27u sock 0,8 0t0 40826406 protocol: TCP
k3s 3477252 root 28u sock 0,8 0t0 40826404 protocol: TCP
k3s 3477252 root 29w FIFO 0,13 0t0 40826407 pipe
k3s 3477252 root 30r FIFO 0,13 0t0 40826408 pipe
k3s 3477252 root 31w FIFO 0,13 0t0 40826408 pipe
k3s 3477252 root 32u sock 0,8 0t0 40839283 protocol: TCP
k3s 3477252 root 33u sock 0,8 0t0 40833415 protocol: TCP
k3s 3477252 root 34u sock 0,8 0t0 40817656 protocol: TCP
k3s 3477252 root 35u sock 0,8 0t0 40833417 protocol: TCP
k3s 3477252 root 36r FIFO 0,13 0t0 40833418 pipe
k3s 3477252 root 37w FIFO 0,13 0t0 40833418 pipe
k3s 3477252 root 38r FIFO 0,13 0t0 40833419 pipe
k3s 3477252 root 39w FIFO 0,13 0t0 40833419 pipe
k3s 3477252 root 40u sock 0,8 0t0 197163985 protocol: TCP
k3s 3477252 root 41u sock 0,8 0t0 40826416 protocol: TCP
k3s 3477252 root 42u sock 0,8 0t0 197171882 protocol: TCP
k3s 3477252 root 43u sock 0,8 0t0 197174540 protocol: TCP
k3s 3477252 root 44r FIFO 0,13 0t0 197174541 pipe
k3s 3477252 root 46w FIFO 0,13 0t0 197174541 pipe
k3s 3477252 root 47r FIFO 0,13 0t0 197161811 pipe
k3s 3477252 root 48u sock 0,8 0t0 40840220 protocol: TCP
k3s 3477252 root 49r FIFO 0,13 0t0 40824358 pipe
k3s 3477252 root 50u sock 0,8 0t0 40817658 protocol: TCP
k3s 3477252 root 51u sock 0,8 0t0 40830112 protocol: TCP
k3s 3477252 root 52u sock 0,8 0t0 40824357 protocol: TCP
k3s 3477252 root 53u sock 0,8 0t0 40827371 protocol: TCP
k3s 3477252 root 54u sock 0,8 0t0 40827307 protocol: TCP
k3s 3477252 root 55w FIFO 0,13 0t0 40824358 pipe
k3s 3477252 root 56u sock 0,8 0t0 118126124 protocol: TCP
k3s 3477252 root 57r FIFO 0,13 0t0 40831219 pipe
k3s 3477252 root 58w FIFO 0,13 0t0 40831219 pipe
k3s 3477252 root 59u sock 0,8 0t0 118143294 protocol: TCP
k3s 3477252 root 60u sock 0,8 0t0 118159733 protocol: TCP
k3s 3477252 root 61u sock 0,8 0t0 118185217 protocol: TCP
k3s 3477252 root 62w FIFO 0,13 0t0 197161811 pipe
k3s 3477252 root 65u sock 0,8 0t0 40838224 protocol: TCP
k3s 3477252 root 66u sock 0,8 0t0 40839297 protocol: TCP
k3s 3477252 root 67u sock 0,8 0t0 40839300 protocol: TCP
k3s 3477252 root 68u sock 0,8 0t0 40839298 protocol: TCP
k3s 3477252 root 69u sock 0,8 0t0 40837236 protocol: TCP
k3s 3477252 root 70r FIFO 0,13 0t0 40830113 pipe
k3s 3477252 root 71w FIFO 0,13 0t0 40830113 pipe
k3s 3477252 root 72r FIFO 0,13 0t0 40839301 pipe
k3s 3477252 root 73w FIFO 0,13 0t0 40839301 pipe
k3s 3477252 root 74r FIFO 0,13 0t0 40837237 pipe
k3s 3477252 root 75w FIFO 0,13 0t0 40837237 pipe
k3s 3477252 root 76r FIFO 0,13 0t0 40841275 pipe
k3s 3477252 root 77w FIFO 0,13 0t0 40841275 pipe
k3s 3477252 root 78u sock 0,8 0t0 40827372 protocol: TCP
k3s 3477252 root 79u sock 0,8 0t0 40827373 protocol: TCP
k3s 3477252 root 80u sock 0,8 0t0 40827374 protocol: TCP
k3s 3477252 root 81u sock 0,8 0t0 40827375 protocol: TCP
k3s 3477252 root 82u sock 0,8 0t0 40827376 protocol: TCP
k3s 3477252 root 83u sock 0,8 0t0 40827377 protocol: TCP
k3s 3477252 root 84u sock 0,8 0t0 40827378 protocol: TCP
k3s 3477252 root 85u sock 0,8 0t0 40827379 protocol: TCP
k3s 3477252 root 86u sock 0,8 0t0 40827380 protocol: TCP
k3s 3477252 root 87u sock 0,8 0t0 40827381 protocol: TCP
k3s 3477252 root 88u sock 0,8 0t0 40827382 protocol: TCP
k3s 3477252 root 89u sock 0,8 0t0 40827383 protocol: TCP
k3s 3477252 root 90u sock 0,8 0t0 40827384 protocol: TCP
k3s 3477252 root 91u sock 0,8 0t0 40827385 protocol: TCP
k3s 3477252 root 92u sock 0,8 0t0 40827386 protocol: TCP
k3s 3477252 root 93u sock 0,8 0t0 40827387 protocol: TCP
k3s 3477252 root 94u sock 0,8 0t0 40827388 protocol: TCP
k3s 3477252 root 95u sock 0,8 0t0 40827389 protocol: TCP
k3s 3477252 root 96u sock 0,8 0t0 40827390 protocol: TCP
k3s 3477252 root 97u sock 0,8 0t0 40827391 protocol: TCP
k3s 3477252 root 98u sock 0,8 0t0 40827392 protocol: TCP
k3s 3477252 root 99u sock 0,8 0t0 40827393 protocol: TCP
k3s 3477252 root 100u sock 0,8 0t0 40827394 protocol: TCP
k3s 3477252 root 101u sock 0,8 0t0 40827395 protocol: TCP
k3s 3477252 root 102u sock 0,8 0t0 40827396 protocol: TCP
k3s 3477252 root 103u sock 0,8 0t0 40827397 protocol: TCP
k3s 3477252 root 104u sock 0,8 0t0 40827398 protocol: TCP
k3s 3477252 root 105u sock 0,8 0t0 40827399 protocol: TCP
k3s 3477252 root 106u sock 0,8 0t0 40827400 protocol: TCP
k3s 3477252 root 107u sock 0,8 0t0 40827401 protocol: TCP
k3s 3477252 root 108u sock 0,8 0t0 40827402 protocol: TCP
k3s 3477252 root 109u sock 0,8 0t0 40827403 protocol: TCP
k3s 3477252 root 110u sock 0,8 0t0 40827404 protocol: TCP
k3s 3477252 root 111u sock 0,8 0t0 40827405 protocol: TCP
k3s 3477252 root 112u sock 0,8 0t0 40830167 protocol: TCP
k3s 3477252 root 113u sock 0,8 0t0 40842640 protocol: UNIX-STREAM
k3s 3477252 root 114u sock 0,8 0t0 40841345 protocol: TCP
k3s 3477252 root 115u sock 0,8 0t0 40841347 protocol: TCP
k3s 3477252 root 116u DIR 0,28 0 154102 /sys/fs/cgroup
k3s 3477252 root 117r CHR 1,11 0t0 187 /dev/kmsg
k3s 3477252 root 118u sock 0,8 0t0 40832679 protocol: UNIX-STREAM
k3s 3477252 root 119r FIFO 0,13 0t0 40841348 pipe
k3s 3477252 root 120r FIFO 0,13 0t0 40830168 pipe
k3s 3477252 root 121w FIFO 0,13 0t0 40841348 pipe
k3s 3477252 root 122w FIFO 0,13 0t0 40830168 pipe
k3s 3477252 root 123u sock 0,8 0t0 40839702 protocol: UNIX-STREAM
k3s 3477252 root 124u a_inode 0,14 0 10579 [eventpoll:125,126]
k3s 3477252 root 125r a_inode 0,14 0 10579 inotify
k3s 3477252 root 126r FIFO 0,13 0t0 40848765 pipe
k3s 3477252 root 127w FIFO 0,13 0t0 40848765 pipe
k3s 3477252 root 128u sock 0,8 0t0 40850524 protocol: TCP
k3s 3477252 root 129u sock 0,8 0t0 40845791 protocol: TCP
k3s 3477252 root 130u sock 0,8 0t0 40835889 protocol: TCP
k3s 3477252 root 131u sock 0,8 0t0 40835891 protocol: TCP
k3s 3477252 root 132u sock 0,8 0t0 40850525 protocol: TCP
k3s 3477252 root 133u sock 0,8 0t0 40850527 protocol: TCP
k3s 3477252 root 134r FIFO 0,13 0t0 40850528 pipe
k3s 3477252 root 135w FIFO 0,13 0t0 40850528 pipe
k3s 3477252 root 136r FIFO 0,13 0t0 40847774 pipe
k3s 3477252 root 137w FIFO 0,13 0t0 40847774 pipe
k3s 3477252 root 138r FIFO 0,13 0t0 40850529 pipe
k3s 3477252 root 139w FIFO 0,13 0t0 40850529 pipe
k3s 3477252 root 140r FIFO 0,13 0t0 40842638 pipe
k3s 3477252 root 141w FIFO 0,13 0t0 40842638 pipe
k3s 3477252 root 142r a_inode 0,14 0 10579 inotify
k3s 3477252 root 143u a_inode 0,14 0 10579 [eventpoll:142,144]
k3s 3477252 root 144r FIFO 0,13 0t0 40835892 pipe
k3s 3477252 root 145w FIFO 0,13 0t0 40835892 pipe
k3s 3477252 root 146u sock 0,8 0t0 40829819 protocol: TCP
k3s 3477252 root 147u sock 0,8 0t0 46349353 protocol: TCP
k3s 3477252 root 148u sock 0,8 0t0 46349354 protocol: TCP
k3s 3477252 root 153u sock 0,8 0t0 199108986 protocol: TCP
k3s 3477252 root 154u sock 0,8 0t0 199117989 protocol: TCP
k3s 3477252 root 155u sock 0,8 0t0 199117991 protocol: TCP
k3s 3477252 root 156u sock 0,8 0t0 117571826 protocol: TCP
k3s 3477252 root 157r FIFO 0,13 0t0 199117992 pipe
k3s 3477252 root 158w FIFO 0,13 0t0 199117992 pipe
k3s 3477252 root 159u sock 0,8 0t0 40836926 protocol: TCPv6
k3s 3477252 root 160r FIFO 0,13 0t0 199108987 pipe
k3s 3477252 root 161u sock 0,8 0t0 118100486 protocol: TCP
k3s 3477252 root 162u sock 0,8 0t0 40845794 protocol: TCP
k3s 3477252 root 163u sock 0,8 0t0 40851698 protocol: UNIX-STREAM
k3s 3477252 root 164u sock 0,8 0t0 40845796 protocol: TCP
k3s 3477252 root 165u sock 0,8 0t0 40830765 protocol: TCP
k3s 3477252 root 166r FIFO 0,13 0t0 40845797 pipe
k3s 3477252 root 167u sock 0,8 0t0 40841842 protocol: TCP
k3s 3477252 root 168w FIFO 0,13 0t0 40845797 pipe
k3s 3477252 root 169w FIFO 0,13 0t0 199108987 pipe
k3s 3477252 root 170r FIFO 0,13 0t0 40847775 pipe
k3s 3477252 root 171w FIFO 0,13 0t0 40847775 pipe
k3s 3477252 root 172u sock 0,8 0t0 40832697 protocol: TCP
k3s 3477252 root 173u sock 0,8 0t0 117639815 protocol: TCP
k3s 3477252 root 174u sock 0,8 0t0 40847776 protocol: TCP
k3s 3477252 root 175r a_inode 0,14 0 10579 inotify
k3s 3477252 root 176u sock 0,8 0t0 117632763 protocol: TCP
k3s 3477252 root 177u sock 0,8 0t0 40847778 protocol: TCP
k3s 3477252 root 178r FIFO 0,13 0t0 40847779 pipe
k3s 3477252 root 179w FIFO 0,13 0t0 40847779 pipe
k3s 3477252 root 180r FIFO 0,13 0t0 40847780 pipe
k3s 3477252 root 181w FIFO 0,13 0t0 40847780 pipe
k3s 3477252 root 182u sock 0,8 0t0 117620724 protocol: TCP
k3s 3477252 root 183r FIFO 0,13 0t0 117632764 pipe
k3s 3477252 root 184w FIFO 0,13 0t0 117632764 pipe
k3s 3477252 root 186r a_inode 0,14 0 10579 inotify
k3s 3477252 root 187r CHR 1,11 0t0 187 /dev/kmsg
k3s 3477252 root 188u sock 0,8 0t0 40852581 protocol: TCP
k3s 3477252 root 189u sock 0,8 0t0 40852583 protocol: TCP
k3s 3477252 root 190r FIFO 0,13 0t0 40851714 pipe
k3s 3477252 root 191u sock 0,8 0t0 40851713 protocol: UNIX-STREAM
k3s 3477252 root 192r a_inode 0,14 0 10579 inotify
k3s 3477252 root 193u a_inode 0,14 0 10579 [eventpoll:190,192]
k3s 3477252 root 194w FIFO 0,13 0t0 40851714 pipe
k3s 3477252 root 195r FIFO 0,13 0t0 40852584 pipe
k3s 3477252 root 196w FIFO 0,13 0t0 40852584 pipe
k3s 3477252 root 197u sock 0,8 0t0 40843700 protocol: TCP
k3s 3477252 root 198r FIFO 0,13 0t0 40852585 pipe
k3s 3477252 root 199w FIFO 0,13 0t0 40852585 pipe
k3s 3477252 root 200u sock 0,8 0t0 40829841 protocol: TCP
k3s 3477252 root 201u sock 0,8 0t0 40834945 protocol: TCP
k3s 3477252 root 202u sock 0,8 0t0 40834947 protocol: TCP
k3s 3477252 root 203r FIFO 0,13 0t0 40834948 pipe
k3s 3477252 root 204w FIFO 0,13 0t0 40834948 pipe
k3s 3477252 root 205r FIFO 0,13 0t0 40842670 pipe
k3s 3477252 root 206w FIFO 0,13 0t0 40842670 pipe
k3s 3477252 root 207u sock 0,8 0t0 40830799 protocol: TCP
k3s 3477252 root 208u sock 0,8 0t0 40830801 protocol: TCP
k3s 3477252 root 209u sock 0,8 0t0 40835004 protocol: TCP
k3s 3477252 root 210u sock 0,8 0t0 40833934 protocol: TCP
k3s 3477252 root 211u sock 0,8 0t0 40833935 protocol: TCP
k3s 3477252 root 212r FIFO 0,13 0t0 40830802 pipe
k3s 3477252 root 213w FIFO 0,13 0t0 40830802 pipe
k3s 3477252 root 214u sock 0,8 0t0 40839855 protocol: TCP
k3s 3477252 root 215u sock 0,8 0t0 40833936 protocol: TCP
k3s 3477252 root 216r FIFO 0,13 0t0 40830803 pipe
k3s 3477252 root 217w FIFO 0,13 0t0 40830803 pipe
k3s 3477252 root 218u sock 0,8 0t0 40833938 protocol: TCP
k3s 3477252 root 219u sock 0,8 0t0 40830860 protocol: TCP
k3s 3477252 root 220u sock 0,8 0t0 40839080 protocol: TCP
k3s 3477252 root 221u sock 0,8 0t0 40851812 protocol: TCP
k3s 3477252 root 222u sock 0,8 0t0 40847844 protocol: TCP
k3s 3477252 root 223u sock 0,8 0t0 40835009 protocol: TCP
k3s 3477252 root 224u sock 0,8 0t0 40835005 protocol: TCP
k3s 3477252 root 225u sock 0,8 0t0 40835006 protocol: TCP
k3s 3477252 root 226u sock 0,8 0t0 40835010 protocol: TCP
k3s 3477252 root 227u sock 0,8 0t0 40835008 protocol: TCP
k3s 3477252 root 228r FIFO 0,13 0t0 40833939 pipe
k3s 3477252 root 229w FIFO 0,13 0t0 40833939 pipe
k3s 3477252 root 230r FIFO 0,13 0t0 40852698 pipe
k3s 3477252 root 231w FIFO 0,13 0t0 40852698 pipe
k3s 3477252 root 232r FIFO 0,13 0t0 40835015 pipe
k3s 3477252 root 233u sock 0,8 0t0 40833940 protocol: TCP
k3s 3477252 root 234u sock 0,8 0t0 40835012 protocol: TCP
k3s 3477252 root 235r FIFO 0,13 0t0 40839081 pipe
k3s 3477252 root 236w FIFO 0,13 0t0 40839081 pipe
k3s 3477252 root 237r FIFO 0,13 0t0 40830861 pipe
k3s 3477252 root 238w FIFO 0,13 0t0 40830861 pipe
k3s 3477252 root 239r FIFO 0,13 0t0 40833941 pipe
k3s 3477252 root 240w FIFO 0,13 0t0 40833941 pipe
k3s 3477252 root 241r FIFO 0,13 0t0 40830862 pipe
k3s 3477252 root 242w FIFO 0,13 0t0 40830862 pipe
k3s 3477252 root 243u sock 0,8 0t0 40852699 protocol: TCP
k3s 3477252 root 244r FIFO 0,13 0t0 40835013 pipe
k3s 3477252 root 245w FIFO 0,13 0t0 40835013 pipe
k3s 3477252 root 246r FIFO 0,13 0t0 40835014 pipe
k3s 3477252 root 247w FIFO 0,13 0t0 40835014 pipe
k3s 3477252 root 248r FIFO 0,13 0t0 40852700 pipe
k3s 3477252 root 249w FIFO 0,13 0t0 40852700 pipe
k3s 3477252 root 250w FIFO 0,13 0t0 40835015 pipe
k3s 3477252 root 251r FIFO 0,13 0t0 40847845 pipe
k3s 3477252 root 252w FIFO 0,13 0t0 40847845 pipe
k3s 3477252 root 253u sock 0,8 0t0 40839873 protocol: TCP
k3s 3477252 root 254r FIFO 0,13 0t0 40847846 pipe
k3s 3477252 root 255w FIFO 0,13 0t0 40847846 pipe
k3s 3477252 root 256u sock 0,8 0t0 40856584 protocol: TCP
k3s 3477252 root 257u sock 0,8 0t0 40856586 protocol: TCP
k3s 3477252 root 258r FIFO 0,13 0t0 40856587 pipe
k3s 3477252 root 259w FIFO 0,13 0t0 40856587 pipe
k3s 3477252 root 260r FIFO 0,13 0t0 40839874 pipe
k3s 3477252 root 261w FIFO 0,13 0t0 40839874 pipe
k3s 3477252 root 262u sock 0,8 0t0 40852799 protocol: TCP
k3s 3477252 root 263u sock 0,8 0t0 40833960 protocol: TCP
k3s 3477252 root 264u sock 0,8 0t0 40833962 protocol: TCP
k3s 3477252 root 265r FIFO 0,13 0t0 40833963 pipe
k3s 3477252 root 266w FIFO 0,13 0t0 40833963 pipe
k3s 3477252 root 267r FIFO 0,13 0t0 40852800 pipe
k3s 3477252 root 268w FIFO 0,13 0t0 40852800 pipe
k3s 3477252 root 269u sock 0,8 0t0 40842060 protocol: TCP
k3s 3477252 root 270u sock 0,8 0t0 40847993 protocol: TCP
k3s 3477252 root 271u sock 0,8 0t0 40847995 protocol: TCP
k3s 3477252 root 272r FIFO 0,13 0t0 40847996 pipe
k3s 3477252 root 273w FIFO 0,13 0t0 40847996 pipe
k3s 3477252 root 274r FIFO 0,13 0t0 40847997 pipe
k3s 3477252 root 275w FIFO 0,13 0t0 40847997 pipe
k3s 3477252 root 276u sock 0,8 0t0 40851966 protocol: TCP
k3s 3477252 root 277u sock 0,8 0t0 40862764 protocol: TCP
k3s 3477252 root 278u sock 0,8 0t0 40862766 protocol: TCP
k3s 3477252 root 279r FIFO 0,13 0t0 40862767 pipe
k3s 3477252 root 280w FIFO 0,13 0t0 40862767 pipe
k3s 3477252 root 281r FIFO 0,13 0t0 40859780 pipe
k3s 3477252 root 282w FIFO 0,13 0t0 40859780 pipe
k3s 3477252 root 283r FIFO 0,13 0t0 117639816 pipe
k3s 3477252 root 284w FIFO 0,13 0t0 117639816 pipe
k3s 3477252 root 290u sock 0,8 0t0 40859781 protocol: TCP
k3s 3477252 root 291u sock 0,8 0t0 40853004 protocol: TCP
k3s 3477252 root 292u sock 0,8 0t0 40853006 protocol: TCP
k3s 3477252 root 293r FIFO 0,13 0t0 40853007 pipe
k3s 3477252 root 294w FIFO 0,13 0t0 40853007 pipe
k3s 3477252 root 295r FIFO 0,13 0t0 40853008 pipe
k3s 3477252 root 296w FIFO 0,13 0t0 40853008 pipe
k3s 3477252 root 297u sock 0,8 0t0 40865805 protocol: TCP
k3s 3477252 root 298r FIFO 0,13 0t0 40853898 pipe
k3s 3477252 root 299u sock 0,8 0t0 40865807 protocol: TCP
k3s 3477252 root 300w FIFO 0,13 0t0 40853898 pipe
k3s 3477252 root 301r FIFO 0,13 0t0 40865808 pipe
k3s 3477252 root 302w FIFO 0,13 0t0 40865808 pipe
k3s 3477252 root 304u sock 0,8 0t0 40860891 protocol: TCP
lsof -a -p (k3s agent 2)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
k3s 3477343 root cwd DIR 0,97 4096 25690623 /
k3s 3477343 root rtd DIR 0,97 4096 25690623 /
k3s 3477343 root txt REG 0,97 130823888 7491470 /bin/containerd
k3s 3477343 root mem REG 259,5 7491470 /bin/containerd (path inode=9490509)
k3s 3477343 root 0u CHR 1,3 0t0 197 /dev/null
k3s 3477343 root 1w FIFO 0,13 0t0 40821350 pipe
k3s 3477343 root 2w FIFO 0,13 0t0 40821351 pipe
k3s 3477343 root 3w CHR 1,3 0t0 197 /dev/null
k3s 3477343 root 4u a_inode 0,14 0 10579 [eventpoll:18,19,20,25,34,35,36,48,66,67,69,71,74,78,105,128,129,130,131,132,139,175,177,179,181,183,211,216,238,239,242,256...]
k3s 3477343 root 5r FIFO 0,13 0t0 40831040 pipe
k3s 3477343 root 6w FIFO 0,13 0t0 40831040 pipe
k3s 3477343 root 7u sock 0,8 0t0 40831044 protocol: UNIX
k3s 3477343 root 8u sock 0,8 0t0 40827053 protocol: TCP
k3s 3477343 root 9u sock 0,8 0t0 40818557 protocol: TCP
k3s 3477343 root 10u sock 0,8 0t0 40827058 protocol: TCP
k3s 3477343 root 11u sock 0,8 0t0 40827060 protocol: TCP
k3s 3477343 root 12r FIFO 0,13 0t0 40827061 pipe
k3s 3477343 root 13w FIFO 0,13 0t0 40827061 pipe
k3s 3477343 root 14r FIFO 0,13 0t0 40818558 pipe
k3s 3477343 root 15w FIFO 0,13 0t0 40818558 pipe
k3s 3477343 root 16w REG 259,5 26047373 23208654 /var/lib/rancher/k3s/agent/containerd/containerd.log
k3s 3477343 root 17r FIFO 0,13 0t0 40833195 pipe
k3s 3477343 root 18u sock 0,8 0t0 40834465 protocol: TCP
k3s 3477343 root 19u sock 0,8 0t0 40833403 protocol: TCP
k3s 3477343 root 20u sock 0,8 0t0 40833405 protocol: TCP
k3s 3477343 root 21r FIFO 0,13 0t0 40833406 pipe
k3s 3477343 root 22w FIFO 0,13 0t0 40833406 pipe
k3s 3477343 root 23r FIFO 0,13 0t0 40826401 pipe
k3s 3477343 root 24w FIFO 0,13 0t0 40826401 pipe
k3s 3477343 root 25u sock 0,8 0t0 40834466 protocol: TCP
k3s 3477343 root 26u sock 0,8 0t0 40823483 protocol: TCP
k3s 3477343 root 27u sock 0,8 0t0 40832390 protocol: TCP
k3s 3477343 root 28u sock 0,8 0t0 117555136 protocol: TCP
k3s 3477343 root 29u sock 0,8 0t0 40823485 protocol: TCP
k3s 3477343 root 30r FIFO 0,13 0t0 40823486 pipe
k3s 3477343 root 31w FIFO 0,13 0t0 40823486 pipe
k3s 3477343 root 32r FIFO 0,13 0t0 40840212 pipe
k3s 3477343 root 33w FIFO 0,13 0t0 40840212 pipe
k3s 3477343 root 34u sock 0,8 0t0 196214192 protocol: TCP
k3s 3477343 root 35u sock 0,8 0t0 196213219 protocol: TCP
k3s 3477343 root 36u sock 0,8 0t0 196213221 protocol: TCP
k3s 3477343 root 37r FIFO 0,13 0t0 196213222 pipe
k3s 3477343 root 38w FIFO 0,13 0t0 196213222 pipe
k3s 3477343 root 39r FIFO 0,13 0t0 196202843 pipe
k3s 3477343 root 40u sock 0,8 0t0 40837218 protocol: TCP
k3s 3477343 root 41u sock 0,8 0t0 40837219 protocol: TCP
k3s 3477343 root 42r FIFO 0,13 0t0 40825521 pipe
k3s 3477343 root 43u sock 0,8 0t0 40837221 protocol: TCP
k3s 3477343 root 44w FIFO 0,13 0t0 40825521 pipe
k3s 3477343 root 45r FIFO 0,13 0t0 40829160 pipe
k3s 3477343 root 46w FIFO 0,13 0t0 40829160 pipe
k3s 3477343 root 47u sock 0,8 0t0 40819684 protocol: TCP
k3s 3477343 root 48u sock 0,8 0t0 40824359 protocol: TCP
k3s 3477343 root 49u sock 0,8 0t0 46337646 protocol: TCP
k3s 3477343 root 50r FIFO 0,13 0t0 40834479 pipe
k3s 3477343 root 51r FIFO 0,13 0t0 40839290 pipe
k3s 3477343 root 52u sock 0,8 0t0 46337647 protocol: TCP
k3s 3477343 root 53w FIFO 0,13 0t0 40834479 pipe
k3s 3477343 root 54u sock 0,8 0t0 40821610 protocol: TCP
k3s 3477343 root 55w FIFO 0,13 0t0 40839290 pipe
k3s 3477343 root 56u sock 0,8 0t0 40821613 protocol: TCP
k3s 3477343 root 57u sock 0,8 0t0 40839287 protocol: TCP
k3s 3477343 root 58u sock 0,8 0t0 40821611 protocol: TCP
k3s 3477343 root 59u sock 0,8 0t0 40839289 protocol: TCP
k3s 3477343 root 60r FIFO 0,13 0t0 40821614 pipe
k3s 3477343 root 61w FIFO 0,13 0t0 40821614 pipe
k3s 3477343 root 62r FIFO 0,13 0t0 40824361 pipe
k3s 3477343 root 63w FIFO 0,13 0t0 40824361 pipe
k3s 3477343 root 64r FIFO 0,13 0t0 40827312 pipe
k3s 3477343 root 65w FIFO 0,13 0t0 40827312 pipe
k3s 3477343 root 66u sock 0,8 0t0 40841273 protocol: TCP
k3s 3477343 root 67u sock 0,8 0t0 117584178 protocol: TCP
k3s 3477343 root 68u sock 0,8 0t0 40841398 protocol: TCP
k3s 3477343 root 69u sock 0,8 0t0 117594286 protocol: TCP
k3s 3477343 root 70u sock 0,8 0t0 40826421 protocol: TCP
k3s 3477343 root 71u sock 0,8 0t0 40826420 protocol: TCP
k3s 3477343 root 72r FIFO 0,13 0t0 40822766 pipe
k3s 3477343 root 73w FIFO 0,13 0t0 40822766 pipe
k3s 3477343 root 74u sock 0,8 0t0 40838222 protocol: TCP
k3s 3477343 root 75u sock 0,8 0t0 40836313 protocol: TCP
k3s 3477343 root 76r FIFO 0,13 0t0 117594289 pipe
k3s 3477343 root 77u sock 0,8 0t0 40811497 protocol: TCP
k3s 3477343 root 78u sock 0,8 0t0 117594288 protocol: TCP
k3s 3477343 root 79u sock 0,8 0t0 40834478 protocol: TCP
k3s 3477343 root 80w FIFO 0,13 0t0 196202843 pipe
k3s 3477343 root 81w FIFO 0,13 0t0 117594289 pipe
k3s 3477343 root 82r FIFO 0,13 0t0 117584180 pipe
k3s 3477343 root 84w FIFO 0,13 0t0 117584180 pipe
k3s 3477343 root 85u sock 0,8 0t0 40841399 protocol: TCP
k3s 3477343 root 89r FIFO 0,13 0t0 40822767 pipe
k3s 3477343 root 90w FIFO 0,13 0t0 40822767 pipe
k3s 3477343 root 91r FIFO 0,13 0t0 40822768 pipe
k3s 3477343 root 92w FIFO 0,13 0t0 40822768 pipe
k3s 3477343 root 96u sock 0,8 0t0 40841400 protocol: TCP
k3s 3477343 root 97u sock 0,8 0t0 40841401 protocol: TCP
k3s 3477343 root 98u sock 0,8 0t0 40841402 protocol: TCP
k3s 3477343 root 99u sock 0,8 0t0 40841403 protocol: TCP
k3s 3477343 root 100u sock 0,8 0t0 40841404 protocol: TCP
k3s 3477343 root 101u sock 0,8 0t0 40841405 protocol: TCP
k3s 3477343 root 105u sock 0,8 0t0 118187211 protocol: TCP
k3s 3477343 root 106u sock 0,8 0t0 40841406 protocol: TCP
k3s 3477343 root 107u sock 0,8 0t0 40841407 protocol: TCP
k3s 3477343 root 108u sock 0,8 0t0 40841408 protocol: TCP
k3s 3477343 root 109u sock 0,8 0t0 40841409 protocol: TCP
k3s 3477343 root 110u sock 0,8 0t0 40841410 protocol: TCP
k3s 3477343 root 111u sock 0,8 0t0 40841411 protocol: TCP
k3s 3477343 root 112u sock 0,8 0t0 40841412 protocol: TCP
k3s 3477343 root 113u sock 0,8 0t0 40841413 protocol: TCP
k3s 3477343 root 114u sock 0,8 0t0 40841414 protocol: TCP
k3s 3477343 root 115u sock 0,8 0t0 40841415 protocol: TCP
k3s 3477343 root 116u sock 0,8 0t0 40841416 protocol: TCP
k3s 3477343 root 117u sock 0,8 0t0 40841417 protocol: TCP
k3s 3477343 root 118u sock 0,8 0t0 40841418 protocol: TCP
k3s 3477343 root 119u sock 0,8 0t0 40841419 protocol: TCP
k3s 3477343 root 120u sock 0,8 0t0 40841420 protocol: TCP
k3s 3477343 root 121u sock 0,8 0t0 40841421 protocol: TCP
k3s 3477343 root 122u sock 0,8 0t0 40841422 protocol: TCP
k3s 3477343 root 123u sock 0,8 0t0 40841423 protocol: TCP
k3s 3477343 root 124u sock 0,8 0t0 40841424 protocol: TCP
k3s 3477343 root 125u sock 0,8 0t0 40841425 protocol: TCP
k3s 3477343 root 126u sock 0,8 0t0 40841426 protocol: TCP
k3s 3477343 root 127u sock 0,8 0t0 40841427 protocol: TCP
k3s 3477343 root 128u sock 0,8 0t0 40841428 protocol: TCP
k3s 3477343 root 129u sock 0,8 0t0 40845356 protocol: TCP
k3s 3477343 root 130u sock 0,8 0t0 40845357 protocol: TCP
k3s 3477343 root 131u sock 0,8 0t0 40845358 protocol: TCP
k3s 3477343 root 132u sock 0,8 0t0 40845359 protocol: TCP
k3s 3477343 root 133u sock 0,8 0t0 40842666 protocol: TCP
k3s 3477343 root 134u DIR 0,28 0 154171 /sys/fs/cgroup
k3s 3477343 root 135u sock 0,8 0t0 40838094 protocol: UNIX-STREAM
k3s 3477343 root 136u sock 0,8 0t0 40850504 protocol: UNIX-STREAM
k3s 3477343 root 137r FIFO 0,13 0t0 40828817 pipe
k3s 3477343 root 138r a_inode 0,14 0 10579 inotify
k3s 3477343 root 139r CHR 1,11 0t0 187 /dev/kmsg
k3s 3477343 root 140u a_inode 0,14 0 10579 [eventpoll:137,138]
k3s 3477343 root 141w FIFO 0,13 0t0 40828817 pipe
k3s 3477343 root 142u sock 0,8 0t0 40835883 protocol: TCP
k3s 3477343 root 143u sock 0,8 0t0 40850505 protocol: TCP
k3s 3477343 root 144u sock 0,8 0t0 40840854 protocol: TCP
k3s 3477343 root 145u sock 0,8 0t0 40834919 protocol: TCP
k3s 3477343 root 146u sock 0,8 0t0 40850507 protocol: TCP
k3s 3477343 root 147u sock 0,8 0t0 40850509 protocol: TCP
k3s 3477343 root 148r FIFO 0,13 0t0 40840855 pipe
k3s 3477343 root 149w FIFO 0,13 0t0 40840855 pipe
k3s 3477343 root 150r FIFO 0,13 0t0 40852547 pipe
k3s 3477343 root 151w FIFO 0,13 0t0 40852547 pipe
k3s 3477343 root 152r FIFO 0,13 0t0 40850510 pipe
k3s 3477343 root 153w FIFO 0,13 0t0 40850510 pipe
k3s 3477343 root 154r FIFO 0,13 0t0 40852548 pipe
k3s 3477343 root 155w FIFO 0,13 0t0 40852548 pipe
k3s 3477343 root 156r a_inode 0,14 0 10579 inotify
k3s 3477343 root 157u a_inode 0,14 0 10579 [eventpoll:156,158]
k3s 3477343 root 158r FIFO 0,13 0t0 40840856 pipe
k3s 3477343 root 159w FIFO 0,13 0t0 40840856 pipe
k3s 3477343 root 160u sock 0,8 0t0 40846994 protocol: TCP
k3s 3477343 root 161u sock 0,8 0t0 40828861 protocol: TCP
k3s 3477343 root 162u sock 0,8 0t0 40840857 protocol: UNIX-STREAM
k3s 3477343 root 163u sock 0,8 0t0 40834920 protocol: TCP
k3s 3477343 root 165u sock 0,8 0t0 40848763 protocol: TCP
k3s 3477343 root 167u sock 0,8 0t0 40848761 protocol: TCP
k3s 3477343 root 170r FIFO 0,13 0t0 40848764 pipe
k3s 3477343 root 171w FIFO 0,13 0t0 40848764 pipe
k3s 3477343 root 172r FIFO 0,13 0t0 40852550 pipe
k3s 3477343 root 173w FIFO 0,13 0t0 40852550 pipe
k3s 3477343 root 175u sock 0,8 0t0 118103426 protocol: TCP
k3s 3477343 root 176u sock 0,8 0t0 40838101 protocol: TCPv6
k3s 3477343 root 177u sock 0,8 0t0 118122901 protocol: TCP
k3s 3477343 root 179u sock 0,8 0t0 40840875 protocol: TCP
k3s 3477343 root 181u sock 0,8 0t0 118145205 protocol: TCP
k3s 3477343 root 182u sock 0,8 0t0 40850515 protocol: TCP
k3s 3477343 root 183u sock 0,8 0t0 40839696 protocol: TCP
k3s 3477343 root 184u sock 0,8 0t0 40850513 protocol: TCP
k3s 3477343 root 187r FIFO 0,13 0t0 40850516 pipe
k3s 3477343 root 188w FIFO 0,13 0t0 40850516 pipe
k3s 3477343 root 189r FIFO 0,13 0t0 40842627 pipe
k3s 3477343 root 190w FIFO 0,13 0t0 40842627 pipe
k3s 3477343 root 193u sock 0,8 0t0 40850519 protocol: UNIX-STREAM
k3s 3477343 root 194r a_inode 0,14 0 10579 inotify
k3s 3477343 root 195u sock 0,8 0t0 40828857 protocol: TCP
k3s 3477343 root 196u sock 0,8 0t0 118146935 protocol: TCP
k3s 3477343 root 197u sock 0,8 0t0 40847021 protocol: UNIX-STREAM
k3s 3477343 root 198r a_inode 0,14 0 10579 inotify
k3s 3477343 root 201r a_inode 0,14 0 10579 inotify
k3s 3477343 root 202r CHR 1,11 0t0 187 /dev/kmsg
k3s 3477343 root 207u a_inode 0,14 0 10579 [eventpoll:198,208]
k3s 3477343 root 208r FIFO 0,13 0t0 40847022 pipe
k3s 3477343 root 209u sock 0,8 0t0 40852632 protocol: TCP
k3s 3477343 root 210w FIFO 0,13 0t0 40847022 pipe
k3s 3477343 root 211u sock 0,8 0t0 40835910 protocol: TCP
k3s 3477343 root 212r FIFO 0,13 0t0 40835911 pipe
k3s 3477343 root 213w FIFO 0,13 0t0 40835911 pipe
k3s 3477343 root 214r FIFO 0,13 0t0 40839006 pipe
k3s 3477343 root 215w FIFO 0,13 0t0 40839006 pipe
k3s 3477343 root 216u sock 0,8 0t0 40847788 protocol: TCP
k3s 3477343 root 217r FIFO 0,13 0t0 40847789 pipe
k3s 3477343 root 218w FIFO 0,13 0t0 40847789 pipe
k3s 3477343 root 219r FIFO 0,13 0t0 40842667 pipe
k3s 3477343 root 220w FIFO 0,13 0t0 40842667 pipe
k3s 3477343 root 221u sock 0,8 0t0 40850614 protocol: TCP
k3s 3477343 root 222u sock 0,8 0t0 40845900 protocol: TCP
k3s 3477343 root 223u sock 0,8 0t0 40845902 protocol: TCP
k3s 3477343 root 224u sock 0,8 0t0 40845914 protocol: TCP
k3s 3477343 root 226u sock 0,8 0t0 40851806 protocol: TCP
k3s 3477343 root 227r FIFO 0,13 0t0 40835981 pipe
k3s 3477343 root 228r FIFO 0,13 0t0 40845903 pipe
k3s 3477343 root 229w FIFO 0,13 0t0 40845903 pipe
k3s 3477343 root 230w FIFO 0,13 0t0 40835981 pipe
k3s 3477343 root 231u sock 0,8 0t0 40830857 protocol: TCP
k3s 3477343 root 233u sock 0,8 0t0 40849689 protocol: TCP
k3s 3477343 root 234w FIFO 0,13 0t0 40830859 pipe
k3s 3477343 root 236u sock 0,8 0t0 40851808 protocol: TCP
k3s 3477343 root 237u sock 0,8 0t0 40851809 protocol: TCP
k3s 3477343 root 238u sock 0,8 0t0 40834999 protocol: TCP
k3s 3477343 root 239u sock 0,8 0t0 40847840 protocol: TCP
k3s 3477343 root 240u sock 0,8 0t0 40851811 protocol: TCP
k3s 3477343 root 242u sock 0,8 0t0 40844862 protocol: TCP
k3s 3477343 root 245u sock 0,8 0t0 40835982 protocol: TCP
k3s 3477343 root 246r FIFO 0,13 0t0 40847841 pipe
k3s 3477343 root 249r FIFO 0,13 0t0 40830858 pipe
k3s 3477343 root 250w FIFO 0,13 0t0 40830858 pipe
k3s 3477343 root 251w FIFO 0,13 0t0 40847841 pipe
k3s 3477343 root 252r FIFO 0,13 0t0 40837030 pipe
k3s 3477343 root 253w FIFO 0,13 0t0 40837030 pipe
k3s 3477343 root 254r FIFO 0,13 0t0 40835983 pipe
k3s 3477343 root 255r FIFO 0,13 0t0 40830859 pipe
k3s 3477343 root 256u sock 0,8 0t0 40855629 protocol: TCP
k3s 3477343 root 257u sock 0,8 0t0 40843862 protocol: TCP
k3s 3477343 root 258r FIFO 0,13 0t0 40844863 pipe
k3s 3477343 root 259w FIFO 0,13 0t0 40844863 pipe
k3s 3477343 root 262r FIFO 0,13 0t0 40849690 pipe
k3s 3477343 root 264u sock 0,8 0t0 40843864 protocol: TCP
k3s 3477343 root 266r FIFO 0,13 0t0 40853555 pipe
k3s 3477343 root 267w FIFO 0,13 0t0 40849690 pipe
k3s 3477343 root 268w FIFO 0,13 0t0 40835983 pipe
k3s 3477343 root 269r FIFO 0,13 0t0 40843865 pipe
k3s 3477343 root 270w FIFO 0,13 0t0 40843865 pipe
k3s 3477343 root 272w FIFO 0,13 0t0 40853555 pipe
k3s 3477343 root 274r FIFO 0,13 0t0 40855630 pipe
k3s 3477343 root 275w FIFO 0,13 0t0 40855630 pipe
lsof -a -p (k3s agent 3)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
k3s 3477415 root cwd DIR 0,96 4096 25690576 /
k3s 3477415 root rtd DIR 0,96 4096 25690576 /
k3s 3477415 root txt REG 0,96 130823888 7491470 /bin/containerd
k3s 3477415 root mem REG 259,5 7491470 /bin/containerd (path inode=9490509)
k3s 3477415 root 0u CHR 1,3 0t0 197 /dev/null
k3s 3477415 root 1w FIFO 0,13 0t0 40814580 pipe
k3s 3477415 root 2w FIFO 0,13 0t0 40814581 pipe
k3s 3477415 root 3w CHR 1,3 0t0 197 /dev/null
k3s 3477415 root 4u a_inode 0,14 0 10579 [eventpoll:5,10,11,40,42,43,47,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,149,263,265...]
k3s 3477415 root 5r FIFO 0,13 0t0 40824243 pipe
k3s 3477415 root 6w FIFO 0,13 0t0 40824243 pipe
k3s 3477415 root 7u sock 0,8 0t0 40819584 protocol: UNIX
k3s 3477415 root 8u sock 0,8 0t0 40825276 protocol: TCP
k3s 3477415 root 9u sock 0,8 0t0 40811399 protocol: TCP
k3s 3477415 root 10u sock 0,8 0t0 40829016 protocol: TCP
k3s 3477415 root 11u sock 0,8 0t0 40829018 protocol: TCP
k3s 3477415 root 12r FIFO 0,13 0t0 40829019 pipe
k3s 3477415 root 13w FIFO 0,13 0t0 40829019 pipe
k3s 3477415 root 14r FIFO 0,13 0t0 40812413 pipe
k3s 3477415 root 15w FIFO 0,13 0t0 40812413 pipe
k3s 3477415 root 16w REG 259,5 25861408 23208636 /var/lib/rancher/k3s/agent/containerd/containerd.log
k3s 3477415 root 17r FIFO 0,13 0t0 40821551 pipe
k3s 3477415 root 18u sock 0,8 0t0 40839272 protocol: TCP
k3s 3477415 root 19u sock 0,8 0t0 40826395 protocol: TCP
k3s 3477415 root 20u sock 0,8 0t0 40826397 protocol: TCP
k3s 3477415 root 21r FIFO 0,13 0t0 40826398 pipe
k3s 3477415 root 22w FIFO 0,13 0t0 40826398 pipe
k3s 3477415 root 23r FIFO 0,13 0t0 40828182 pipe
k3s 3477415 root 24w FIFO 0,13 0t0 40828182 pipe
k3s 3477415 root 25u sock 0,8 0t0 40828186 protocol: TCP
k3s 3477415 root 26u sock 0,8 0t0 40822759 protocol: TCP
k3s 3477415 root 27u sock 0,8 0t0 40828189 protocol: TCP
k3s 3477415 root 28u sock 0,8 0t0 40819680 protocol: TCP
k3s 3477415 root 29u sock 0,8 0t0 40828196 protocol: TCP
k3s 3477415 root 30r FIFO 0,13 0t0 40819681 pipe
k3s 3477415 root 31w FIFO 0,13 0t0 40819681 pipe
k3s 3477415 root 32u sock 0,8 0t0 40827298 protocol: TCP
k3s 3477415 root 33r FIFO 0,13 0t0 40830092 pipe
k3s 3477415 root 34w FIFO 0,13 0t0 40830092 pipe
k3s 3477415 root 35u sock 0,8 0t0 40827300 protocol: TCP
k3s 3477415 root 36r FIFO 0,13 0t0 40827301 pipe
k3s 3477415 root 37w FIFO 0,13 0t0 40827301 pipe
k3s 3477415 root 38r FIFO 0,13 0t0 40827302 pipe
k3s 3477415 root 39w FIFO 0,13 0t0 40827302 pipe
k3s 3477415 root 40u sock 0,8 0t0 46336525 protocol: TCP
k3s 3477415 root 41u sock 0,8 0t0 173718248 protocol: TCP
k3s 3477415 root 42u sock 0,8 0t0 46336526 protocol: TCP
k3s 3477415 root 43u sock 0,8 0t0 40838220 protocol: TCP
k3s 3477415 root 45u sock 0,8 0t0 173714217 protocol: TCP
k3s 3477415 root 47u sock 0,8 0t0 173719960 protocol: TCP
k3s 3477415 root 48u sock 0,8 0t0 40827313 protocol: TCP
k3s 3477415 root 49r FIFO 0,13 0t0 173719961 pipe
k3s 3477415 root 50u sock 0,8 0t0 40827310 protocol: TCP
k3s 3477415 root 51w FIFO 0,13 0t0 173719961 pipe
k3s 3477415 root 52u sock 0,8 0t0 40828205 protocol: TCP
k3s 3477415 root 53u sock 0,8 0t0 40826471 protocol: TCP
k3s 3477415 root 54r FIFO 0,13 0t0 173731883 pipe
k3s 3477415 root 55u sock 0,8 0t0 40828207 protocol: TCP
k3s 3477415 root 57r FIFO 0,13 0t0 40828208 pipe
k3s 3477415 root 58w FIFO 0,13 0t0 40828208 pipe
k3s 3477415 root 60u sock 0,8 0t0 196375468 protocol: TCP
k3s 3477415 root 61u sock 0,8 0t0 196398095 protocol: TCP
k3s 3477415 root 62r FIFO 0,13 0t0 40828209 pipe
k3s 3477415 root 63w FIFO 0,13 0t0 40828209 pipe
k3s 3477415 root 64u sock 0,8 0t0 40840232 protocol: TCP
k3s 3477415 root 65u sock 0,8 0t0 40842258 protocol: TCP
k3s 3477415 root 66u sock 0,8 0t0 40826425 protocol: TCP
k3s 3477415 root 67u sock 0,8 0t0 40826426 protocol: TCP
k3s 3477415 root 68u sock 0,8 0t0 40826428 protocol: TCP
k3s 3477415 root 69u sock 0,8 0t0 40842259 protocol: TCP
k3s 3477415 root 70r FIFO 0,13 0t0 40826429 pipe
k3s 3477415 root 71w FIFO 0,13 0t0 40826429 pipe
k3s 3477415 root 72r FIFO 0,13 0t0 40838225 pipe
k3s 3477415 root 73w FIFO 0,13 0t0 40838225 pipe
k3s 3477415 root 74r FIFO 0,13 0t0 40842260 pipe
k3s 3477415 root 75w FIFO 0,13 0t0 40842260 pipe
k3s 3477415 root 76r FIFO 0,13 0t0 40842261 pipe
k3s 3477415 root 77w FIFO 0,13 0t0 40842261 pipe
k3s 3477415 root 78u sock 0,8 0t0 40826472 protocol: TCP
k3s 3477415 root 79u sock 0,8 0t0 40826473 protocol: TCP
k3s 3477415 root 80u sock 0,8 0t0 40826474 protocol: TCP
k3s 3477415 root 81u sock 0,8 0t0 40826475 protocol: TCP
k3s 3477415 root 82u sock 0,8 0t0 40826476 protocol: TCP
k3s 3477415 root 83u sock 0,8 0t0 40826477 protocol: TCP
k3s 3477415 root 84u sock 0,8 0t0 40826478 protocol: TCP
k3s 3477415 root 85u sock 0,8 0t0 40826479 protocol: TCP
k3s 3477415 root 86u sock 0,8 0t0 40826480 protocol: TCP
k3s 3477415 root 87u sock 0,8 0t0 40826481 protocol: TCP
k3s 3477415 root 88u sock 0,8 0t0 40826482 protocol: TCP
k3s 3477415 root 89u sock 0,8 0t0 40826483 protocol: TCP
k3s 3477415 root 90u sock 0,8 0t0 40826484 protocol: TCP
k3s 3477415 root 91u sock 0,8 0t0 40826485 protocol: TCP
k3s 3477415 root 92u sock 0,8 0t0 40826486 protocol: TCP
k3s 3477415 root 93u sock 0,8 0t0 40826487 protocol: TCP
k3s 3477415 root 94u sock 0,8 0t0 40826488 protocol: TCP
k3s 3477415 root 95u sock 0,8 0t0 40826489 protocol: TCP
k3s 3477415 root 96u sock 0,8 0t0 40826490 protocol: TCP
k3s 3477415 root 97u sock 0,8 0t0 40826491 protocol: TCP
k3s 3477415 root 98u sock 0,8 0t0 40826492 protocol: TCP
k3s 3477415 root 99u sock 0,8 0t0 40826493 protocol: TCP
k3s 3477415 root 100u sock 0,8 0t0 40826494 protocol: TCP
k3s 3477415 root 101u sock 0,8 0t0 40826495 protocol: TCP
k3s 3477415 root 102u sock 0,8 0t0 40826496 protocol: TCP
k3s 3477415 root 103u sock 0,8 0t0 40826497 protocol: TCP
k3s 3477415 root 104u sock 0,8 0t0 40826498 protocol: TCP
k3s 3477415 root 105u sock 0,8 0t0 40826499 protocol: TCP
k3s 3477415 root 106u sock 0,8 0t0 40826500 protocol: TCP
k3s 3477415 root 107u sock 0,8 0t0 40826501 protocol: TCP
k3s 3477415 root 108u sock 0,8 0t0 40826502 protocol: TCP
k3s 3477415 root 109u sock 0,8 0t0 40826503 protocol: TCP
k3s 3477415 root 110u sock 0,8 0t0 40826504 protocol: TCP
k3s 3477415 root 111u sock 0,8 0t0 40826505 protocol: TCP
k3s 3477415 root 112u sock 0,8 0t0 40827435 protocol: TCP
k3s 3477415 root 113r FIFO 0,13 0t0 40827438 pipe
k3s 3477415 root 114u sock 0,8 0t0 40827437 protocol: TCP
k3s 3477415 root 115w FIFO 0,13 0t0 40827438 pipe
k3s 3477415 root 116r FIFO 0,13 0t0 40848756 pipe
k3s 3477415 root 117u sock 0,8 0t0 40830169 protocol: TCP
k3s 3477415 root 118u DIR 0,28 0 154240 /sys/fs/cgroup
k3s 3477415 root 119u sock 0,8 0t0 40828801 protocol: UNIX-STREAM
k3s 3477415 root 120r FIFO 0,13 0t0 40827439 pipe
k3s 3477415 root 121w FIFO 0,13 0t0 40827439 pipe
k3s 3477415 root 122u sock 0,8 0t0 40843653 protocol: UNIX-STREAM
k3s 3477415 root 123r CHR 1,11 0t0 187 /dev/kmsg
k3s 3477415 root 124r a_inode 0,14 0 10579 inotify
k3s 3477415 root 125u a_inode 0,14 0 10579 [eventpoll:124,126]
k3s 3477415 root 126r FIFO 0,13 0t0 40828802 pipe
k3s 3477415 root 127w FIFO 0,13 0t0 40828802 pipe
k3s 3477415 root 128u sock 0,8 0t0 40849557 protocol: TCP
k3s 3477415 root 129u sock 0,8 0t0 40849558 protocol: TCP
k3s 3477415 root 130u sock 0,8 0t0 40849560 protocol: TCP
k3s 3477415 root 131u sock 0,8 0t0 40840847 protocol: TCP
k3s 3477415 root 132r a_inode 0,14 0 10579 inotify
k3s 3477415 root 133u a_inode 0,14 0 10579 [eventpoll:132,136]
k3s 3477415 root 134r FIFO 0,13 0t0 40849561 pipe
k3s 3477415 root 135w FIFO 0,13 0t0 40849561 pipe
k3s 3477415 root 136r FIFO 0,13 0t0 40838084 pipe
k3s 3477415 root 137w FIFO 0,13 0t0 40838084 pipe
k3s 3477415 root 138u sock 0,8 0t0 40848755 protocol: TCP
k3s 3477415 root 139u sock 0,8 0t0 40848753 protocol: TCP
k3s 3477415 root 140r FIFO 0,13 0t0 40840848 pipe
k3s 3477415 root 141w FIFO 0,13 0t0 40840848 pipe
k3s 3477415 root 142w FIFO 0,13 0t0 40848756 pipe
k3s 3477415 root 143r FIFO 0,13 0t0 40848757 pipe
k3s 3477415 root 144w FIFO 0,13 0t0 40848757 pipe
k3s 3477415 root 145u sock 0,8 0t0 40828803 protocol: TCP
k3s 3477415 root 148u sock 0,8 0t0 40839871 protocol: TCP
k3s 3477415 root 149u sock 0,8 0t0 117556838 protocol: TCP
k3s 3477415 root 150u sock 0,8 0t0 40842616 protocol: UNIX-STREAM
k3s 3477415 root 151u sock 0,8 0t0 118122109 protocol: TCP
k3s 3477415 root 152w FIFO 0,13 0t0 173731883 pipe
k3s 3477415 root 153u sock 0,8 0t0 118136574 protocol: TCP
k3s 3477415 root 154u sock 0,8 0t0 118184250 protocol: TCP
k3s 3477415 root 156u sock 0,8 0t0 196398097 protocol: TCP
k3s 3477415 root 157r FIFO 0,13 0t0 196398098 pipe
k3s 3477415 root 158u sock 0,8 0t0 40844727 protocol: TCPv6
k3s 3477415 root 159w FIFO 0,13 0t0 196398098 pipe
k3s 3477415 root 160r FIFO 0,13 0t0 196375469 pipe
k3s 3477415 root 161w FIFO 0,13 0t0 196375469 pipe
k3s 3477415 root 163u sock 0,8 0t0 40838089 protocol: TCP
k3s 3477415 root 164r FIFO 0,13 0t0 40850499 pipe
k3s 3477415 root 165u sock 0,8 0t0 40850496 protocol: TCP
k3s 3477415 root 166r FIFO 0,13 0t0 40838090 pipe
k3s 3477415 root 167u sock 0,8 0t0 40838087 protocol: TCP
k3s 3477415 root 168u sock 0,8 0t0 40848759 protocol: TCP
k3s 3477415 root 169u sock 0,8 0t0 40840850 protocol: TCP
k3s 3477415 root 170w FIFO 0,13 0t0 40838090 pipe
k3s 3477415 root 171u sock 0,8 0t0 40850498 protocol: TCP
k3s 3477415 root 172r FIFO 0,13 0t0 40834918 pipe
k3s 3477415 root 173w FIFO 0,13 0t0 40834918 pipe
k3s 3477415 root 174w FIFO 0,13 0t0 40850499 pipe
k3s 3477415 root 175r FIFO 0,13 0t0 40850500 pipe
k3s 3477415 root 176w FIFO 0,13 0t0 40850500 pipe
k3s 3477415 root 177u sock 0,8 0t0 40850502 protocol: UNIX-STREAM
k3s 3477415 root 178r a_inode 0,14 0 10579 inotify
k3s 3477415 root 179u sock 0,8 0t0 40834928 protocol: TCP
k3s 3477415 root 180r a_inode 0,14 0 10579 inotify
k3s 3477415 root 181r CHR 1,11 0t0 187 /dev/kmsg
k3s 3477415 root 182r a_inode 0,14 0 10579 inotify
k3s 3477415 root 183u sock 0,8 0t0 40839002 protocol: TCP
k3s 3477415 root 184u a_inode 0,14 0 10579 [eventpoll:182,189]
k3s 3477415 root 185u sock 0,8 0t0 40841844 protocol: UNIX-STREAM
k3s 3477415 root 186u sock 0,8 0t0 40839004 protocol: TCP
k3s 3477415 root 187u sock 0,8 0t0 40834942 protocol: TCP
k3s 3477415 root 188r FIFO 0,13 0t0 40839005 pipe
k3s 3477415 root 189r FIFO 0,13 0t0 40841845 pipe
k3s 3477415 root 190w FIFO 0,13 0t0 40841845 pipe
k3s 3477415 root 191w FIFO 0,13 0t0 40839005 pipe
k3s 3477415 root 192r FIFO 0,13 0t0 40834929 pipe
k3s 3477415 root 193w FIFO 0,13 0t0 40834929 pipe
k3s 3477415 root 194u sock 0,8 0t0 40847024 protocol: TCP
k3s 3477415 root 195u sock 0,8 0t0 40829827 protocol: TCP
k3s 3477415 root 196u sock 0,8 0t0 40829829 protocol: TCP
k3s 3477415 root 197r FIFO 0,13 0t0 40829830 pipe
k3s 3477415 root 198w FIFO 0,13 0t0 40829830 pipe
k3s 3477415 root 199r FIFO 0,13 0t0 40850545 pipe
k3s 3477415 root 200w FIFO 0,13 0t0 40850545 pipe
k3s 3477415 root 201u sock 0,8 0t0 40852565 protocol: TCP
k3s 3477415 root 202r FIFO 0,13 0t0 40852568 pipe
k3s 3477415 root 203u sock 0,8 0t0 40849686 protocol: TCP
k3s 3477415 root 204w FIFO 0,13 0t0 40852568 pipe
k3s 3477415 root 205u sock 0,8 0t0 40852567 protocol: TCP
k3s 3477415 root 206r FIFO 0,13 0t0 40832696 pipe
k3s 3477415 root 207w FIFO 0,13 0t0 40832696 pipe
k3s 3477415 root 208u sock 0,8 0t0 118092697 protocol: TCP
k3s 3477415 root 209u sock 0,8 0t0 40840960 protocol: TCP
k3s 3477415 root 210u sock 0,8 0t0 40850609 protocol: TCP
k3s 3477415 root 211u sock 0,8 0t0 40847101 protocol: TCP
k3s 3477415 root 212u sock 0,8 0t0 40839078 protocol: TCP
k3s 3477415 root 213u sock 0,8 0t0 40847833 protocol: TCP
k3s 3477415 root 214u sock 0,8 0t0 40847831 protocol: TCP
k3s 3477415 root 215u sock 0,8 0t0 118165651 protocol: TCP
k3s 3477415 root 216u sock 0,8 0t0 40849687 protocol: TCP
k3s 3477415 root 217u sock 0,8 0t0 40852690 protocol: TCP
k3s 3477415 root 218u sock 0,8 0t0 40847834 protocol: TCP
k3s 3477415 root 220u sock 0,8 0t0 40847835 protocol: TCP
k3s 3477415 root 221u sock 0,8 0t0 40847836 protocol: TCP
k3s 3477415 root 222u sock 0,8 0t0 40850610 protocol: TCP
k3s 3477415 root 223u sock 0,8 0t0 40847838 protocol: TCP
k3s 3477415 root 224u sock 0,8 0t0 40839853 protocol: TCP
k3s 3477415 root 225r FIFO 0,13 0t0 40849688 pipe
k3s 3477415 root 226w FIFO 0,13 0t0 40849688 pipe
k3s 3477415 root 227r FIFO 0,13 0t0 40839854 pipe
k3s 3477415 root 228r FIFO 0,13 0t0 40839079 pipe
k3s 3477415 root 229w FIFO 0,13 0t0 40839079 pipe
k3s 3477415 root 230r FIFO 0,13 0t0 40842789 pipe
k3s 3477415 root 231w FIFO 0,13 0t0 40842789 pipe
k3s 3477415 root 232r FIFO 0,13 0t0 40850611 pipe
k3s 3477415 root 233w FIFO 0,13 0t0 40850611 pipe
k3s 3477415 root 238u sock 0,8 0t0 40852789 protocol: TCP
k3s 3477415 root 239r FIFO 0,13 0t0 40852691 pipe
k3s 3477415 root 240r FIFO 0,13 0t0 40850613 pipe
k3s 3477415 root 241w FIFO 0,13 0t0 40839854 pipe
k3s 3477415 root 242w FIFO 0,13 0t0 40852691 pipe
k3s 3477415 root 243w FIFO 0,13 0t0 40850613 pipe
k3s 3477415 root 244r FIFO 0,13 0t0 40847839 pipe
k3s 3477415 root 245w FIFO 0,13 0t0 40847839 pipe
k3s 3477415 root 246r FIFO 0,13 0t0 40837029 pipe
k3s 3477415 root 247w FIFO 0,13 0t0 40837029 pipe
k3s 3477415 root 248r FIFO 0,13 0t0 40830852 pipe
k3s 3477415 root 249w FIFO 0,13 0t0 40830852 pipe
k3s 3477415 root 250u sock 0,8 0t0 40852791 protocol: TCP
k3s 3477415 root 251r FIFO 0,13 0t0 40852792 pipe
k3s 3477415 root 252w FIFO 0,13 0t0 40852792 pipe
k3s 3477415 root 253r FIFO 0,13 0t0 40839872 pipe
k3s 3477415 root 254w FIFO 0,13 0t0 40839872 pipe
k3s 3477415 root 255u sock 0,8 0t0 40847171 protocol: TCP
k3s 3477415 root 256u sock 0,8 0t0 40833956 protocol: TCP
k3s 3477415 root 257u sock 0,8 0t0 40833958 protocol: TCP
k3s 3477415 root 258r FIFO 0,13 0t0 40833959 pipe
k3s 3477415 root 259w FIFO 0,13 0t0 40833959 pipe
k3s 3477415 root 260r FIFO 0,13 0t0 40847172 pipe
k3s 3477415 root 261w FIFO 0,13 0t0 40847172 pipe
k3s 3477415 root 263u sock 0,8 0t0 117637202 protocol: TCP
k3s 3477415 root 264u sock 0,8 0t0 117637204 protocol: TCP
k3s 3477415 root 265u sock 0,8 0t0 117627441 protocol: TCP
k3s 3477415 root 266r FIFO 0,13 0t0 117637205 pipe
k3s 3477415 root 267w FIFO 0,13 0t0 117637205 pipe
k3s 3477415 root 268r FIFO 0,13 0t0 117627442 pipe
k3s 3477415 root 269w FIFO 0,13 0t0 117627442 pipe
Restarting an arbitrary agent seems to have fixed the other cluster, supporting the resource exhaustion hypothesis. Afterwards, however, I tried restarting the whole primary cluster, and got weird resource exhaustion errors:
$ docker restart k3d-k3s-default-{server-0,agent-{0,1,2}}
k3d-k3s-default-server-0
k3d-k3s-default-agent-0
Error response from daemon: Cannot restart container k3d-k3s-default-agent-1: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/tmp/k3d/kubelet/pods" to rootfs at "/var/lib/kubelet/pods": mount /tmp/k3d/kubelet/pods:/var/lib/kubelet/pods (via /proc/self/fd/6), flags: 0x5000: no space left on device: unknown
Error response from daemon: Cannot restart container k3d-k3s-default-agent-2: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/tmp/k3d/kubelet/pods" to rootfs at "/var/lib/kubelet/pods": mount /tmp/k3d/kubelet/pods:/var/lib/kubelet/pods (via /proc/self/fd/6), flags: 0x5000: no space left on device: unknown
However, /tmp has plenty of available space, and the system is nowhere close to running out of RAM:
$ df -h /tmp
Filesystem Size Used Avail Use% Mounted on
tmpfs 32G 29M 32G 1% /tmp
$ free -h
total used free shared buff/cache available
Mem: 62Gi 27Gi 18Gi 1,4Gi 16Gi 33Gi
Swap: 0B 0B 0B
There's also plenty of both space and inodes available on disk:
$ df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p3 914G 658G 215G 76% /
$ df -i /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/nvme0n1p3 60858368 7070237 53788131 12% /
Looks like some mount shenanigans were afoot, mount -l | wc -l returned around 30k entries. Unmounting all of them (by running cat /proc/mounts | awk '{ print $2 }' | grep k3d | xargs sudo umount repeatedly) seems to have got the number of mounts down to 37. This allowed the cluster to start again, bumping it up to 3962 mounts. However, stopping the cluster after that only removed 10 of those mounts. Starting the cluster again got me back up to 31397 mounts, and prevented the third agent from starting up.
This doesn't seem to be mount-related, after all. Getting the same issue now, despite mount | wc -l being stable at 69 entries after having deleted and recreated my main cluster before the weekend.
Seeing this same behaviour in Ubuntu 22.04 - Docker 24.x
docker info
Client: Version: 24.0.2-rd Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.11.0 Path: /home/aitor/.docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.19.0 Path: /home/aitor/.docker/cli-plugins/docker-composeServer: Containers: 7 Running: 2 Paused: 0 Stopped: 5 Images: 29 Server Version: 20.10.25 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux Default Runtime: runc Init Binary: docker-init containerd version: runc version: init version: Security Options: apparmor seccomp Profile: default cgroupns Kernel Version: 6.2.0-26-generic Operating System: KDE neon 5.27 OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 31.07GiB Name: d98f8d3.office.telnyx.com ID: ETTI:43IK:QCL6:NXVE:MWT2:IWG6:O7KC:ZAQ7:MZU3:7RML:JFBN:R5MC Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: true Insecure Registries: 127.0.0.1:5000 k3d-local-registry:5000 Registry Mirrors: http://k3d-local-registry:5000/ http://127.0.0.1:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/16, Size: 24
