ksniff icon indicating copy to clipboard operation
ksniff copied to clipboard

Published 20 hours ago • verified publisher icon eldadru

can not run sniff in k3s cluster

Open houshym opened this issue 3 years ago • 7 comments

I tried to run sniff on my k3s cluster and I am getting the following error

INFO[0000] waiting for pod successful startup INFO[0014] pod: 'ksniff-sfbwk' created successfully on node: 'k8s-worker2' INFO[0014] spawning wireshark! INFO[0014] starting remote sniffing using privileged pod INFO[0014] executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="/run/containerd/containerd.sock" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}" export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT} crictl pull docker.io/maintained/tcpdump:latest >/dev/null netns=$(crictl inspect 977ef1dba9f4d7a5f228c5b1de86a68117a8bf288f7684c450298bc8f3f2ccf4 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"') exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-ofqulVrd tcpdump -i any -U -w - ]' on container: 'ksniff-privileged', pod: 'ksniff-sfbwk', namespace: 'sfs-istio' ERRO[0015] failed executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="/run/containerd/containerd.sock" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}" export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT} crictl pull docker.io/maintained/tcpdump:latest >/dev/null netns=$(crictl inspect 977ef1dba9f4d7a5f228c5b1de86a68117a8bf288f7684c450298bc8f3f2ccf4 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"') exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-ofqulVrd tcpdump -i any -U -w - ]', exitCode: '1', stdErr: '+ export 'CONTAINERD_SOCKET=/run/containerd/containerd.sock'

  • export 'CONTAINERD_NAMESPACE=k8s.io'
  • export 'CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock'
  • export 'IMAGE_SERVICE_ENDPOINT=unix:///host/run/containerd/containerd.sock'
  • crictl pull docker.io/maintained/tcpdump:latest time="2021-07-09T17:54:03Z" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService" ' error="command terminated with exit code 1" ERRO[0015] failed to start sniffing using privileged pod, exit code: '1' error="command terminated with exit code 1" ERRO[0015] failed to start remote sniffing, stopping wireshark error="command terminated with exit code 1" INFO[0015] starting sniffer cleanup INFO[0015] removing privileged container: 'ksniff-privileged' INFO[0015] executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="/run/containerd/containerd.sock" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_ID="ksniff-container-ofqulVrd" chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID} ]' on container: 'ksniff-privileged', pod: 'ksniff-sfbwk', namespace: 'sfs-istio' ERRO[0016] failed executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="/run/containerd/containerd.sock" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_ID="ksniff-container-ofqulVrd" chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID} ]', exitCode: '1', stdErr: '+ export 'CONTAINERD_SOCKET=/run/containerd/containerd.sock'
  • export 'CONTAINERD_NAMESPACE=k8s.io'
  • export 'CONTAINER_ID=ksniff-container-ofqulVrd'
  • chroot /host ctr -a /run/containerd/containerd.sock task kill -s SIGKILL ksniff-container-ofqulVrd ctr: container "ksniff-container-ofqulVrd" in namespace "k8s.io": not found ' error="command terminated with exit code 1" ERRO[0016] failed to remove privileged container: 'ksniff-privileged', exit code: '1', please manually remove it error="command terminated with exit code 1" INFO[0016] removing pod: 'ksniff-sfbwk' INFO[0016] removing privileged pod: 'ksniff-sfbwk' INFO[0016] privileged pod: 'ksniff-sfbwk' removed INFO[0016] pod: 'ksniff-sfbwk' removed successfully INFO[0016] sniffer cleanup completed successfully Error: signal: killed

houshym avatar Jul 09 '21 18:07 houshym

I'm having the same error:

INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'traefik'                
INFO[0000] sniffing method: privileged pod              
INFO[0000] sniffing on pod: 'traefik-658d68cc6b-smh4k' [namespace: 'kube-system', container: 'traefik', filter: '', interface: 'any'] 
INFO[0000] creating privileged pod on node: 'olaf'      
INFO[0000] pod created: &Pod{ObjectMeta:{ksniff-xvmg8 ksniff- kube-system  10f35316-df4a-4664-9973-6c82b105cdd4 2186114 0 2021-08-26 16:21:57 +0200 CEST <nil> <nil> map[app:ksniff] map[] [] []  [{kubectl-sniff Update v1 2021-08-26 16:21:57 +0200 CEST FieldsV1 FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 103 101 110 101 114 97 116 101 78 97 109 101 34 58 123 125 44 34 102 58 108 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 112 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 102 58 99 111 110 116 97 105 110 101 114 115 34 58 123 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 107 115 110 105 102 102 45 112 114 105 118 105 108 101 103 101 100 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 99 111 109 109 97 110 100 34 58 123 125 44 34 102 58 105 109 97 103 101 34 58 123 125 44 34 102 58 105 109 97 103 101 80 117 108 108 80 111 108 105 99 121 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 114 101 115 111 117 114 99 101 115 34 58 123 125 44 34 102 58 115 101 99 117 114 105 116 121 67 111 110 116 101 120 116 34 58 123 34 46 34 58 123 125 44 34 102 58 112 114 105 118 105 108 101 103 101 100 34 58 123 125 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 77 101 115 115 97 103 101 80 97 116 104 34 58 123 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 77 101 115 115 97 103 101 80 111 108 105 99 121 34 58 123 125 44 34 102 58 118 111 108 117 109 101 77 111 117 110 116 115 34 58 123 34 46 34 58 123 125 44 34 107 58 123 92 34 109 111 117 110 116 80 97 116 104 92 34 58 92 34 47 104 111 115 116 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 109 111 117 110 116 80 97 116 104 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 125 44 34 107 58 123 92 34 109 111 117 110 116 80 97 116 104 92 34 58 92 34 47 114 117 110 47 99 111 110 116 97 105 110 101 114 100 47 99 111 110 116 97 105 110 101 114 100 46 115 111 99 107 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 109 111 117 110 116 80 97 116 104 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 114 101 97 100 79 110 108 121 34 58 123 125 125 125 125 125 44 34 102 58 100 110 115 80 111 108 105 99 121 34 58 123 125 44 34 102 58 101 110 97 98 108 101 83 101 114 118 105 99 101 76 105 110 107 115 34 58 123 125 44 34 102 58 104 111 115 116 80 73 68 34 58 123 125 44 34 102 58 110 111 100 101 78 97 109 101 34 58 123 125 44 34 102 58 114 101 115 116 97 114 116 80 111 108 105 99 121 34 58 123 125 44 34 102 58 115 99 104 101 100 117 108 101 114 78 97 109 101 34 58 123 125 44 34 102 58 115 101 99 117 114 105 116 121 67 111 110 116 101 120 116 34 58 123 125 44 34 102 58 116 101 114 109 105 110 97 116 105 111 110 71 114 97 99 101 80 101 114 105 111 100 83 101 99 111 110 100 115 34 58 123 125 44 34 102 58 118 111 108 117 109 101 115 34 58 123 34 46 34 58 123 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 99 111 110 116 97 105 110 101 114 45 115 111 99 107 101 116 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 104 111 115 116 80 97 116 104 34 58 123 34 46 34 58 123 125 44 34 102 58 112 97 116 104 34 58 123 125 44 34 102 58 116 121 112 101 34 58 123 125 125 44 34 102 58 110 97 109 101 34 58 123 125 125 44 34 107 58 123 92 34 110 97 109 101 92 34 58 92 34 104 111 115 116 92 34 125 34 58 123 34 46 34 58 123 125 44 34 102 58 104 111 115 116 80 97 116 104 34 58 123 34 46 34 58 123 125 44 34 102 58 112 97 116 104 34 58 123 125 44 34 102 58 116 121 112 101 34 58 123 125 125 44 34 102 58 110 97 109 101 34 58 123 125 125 125 125 125],}}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:host,VolumeSource:VolumeSource{HostPath:&HostPathVolumeSource{Path:/,Type:*Directory,},EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},Volume{Name:container-socket,VolumeSource:VolumeSource{HostPath:&HostPathVolumeSource{Path:/run/containerd/containerd.sock,Type:*Socket,},EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},Volume{Name:kube-api-access-lvqc4,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:ksniff-privileged,Image:docker.io/hamravesh/ksniff-helper:v3,Command:[sh -c sleep 10000000],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:container-socket,ReadOnly:true,MountPath:/run/containerd/containerd.sock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:host,ReadOnly:false,MountPath:/host,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-lvqc4,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:olaf,HostNetwork:false,HostPID:true,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},} 
INFO[0000] waiting for pod successful startup           
INFO[0002] pod: 'ksniff-xvmg8' created successfully on node: 'olaf' 
INFO[0002] spawning wireshark!                          
INFO[0002] starting remote sniffing using privileged pod 
INFO[0002] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 44a36bd77c0d5ca600bbe00c6a6df9770f3f2c87f8d3f7de8c007d86c0647e46 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-oZQpVNjG tcpdump -i any -U -w -  
    ]' on container: 'ksniff-privileged', pod: 'ksniff-xvmg8', namespace: 'kube-system' 
ERRO[0002] failed executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 44a36bd77c0d5ca600bbe00c6a6df9770f3f2c87f8d3f7de8c007d86c0647e46 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-oZQpVNjG tcpdump -i any -U -w -  
    ]', exitCode: '1', stdErr: '+ export 'CONTAINERD_SOCKET=/run/containerd/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock'
+ export 'IMAGE_SERVICE_ENDPOINT=unix:///host/run/containerd/containerd.sock'
+ crictl pull docker.io/maintained/tcpdump:latest
time="2021-08-26T14:22:00Z" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"
'  error="command terminated with exit code 1"
ERRO[0002] failed to start sniffing using privileged pod, exit code: '1'  error="command terminated with exit code 1"
ERRO[0002] failed to start remote sniffing, stopping wireshark  error="command terminated with exit code 1"
INFO[0002] starting sniffer cleanup                     
INFO[0002] removing privileged container: 'ksniff-privileged' 
INFO[0002] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-oZQpVNjG"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' on container: 'ksniff-privileged', pod: 'ksniff-xvmg8', namespace: 'kube-system' 
ERRO[0002] failed executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-oZQpVNjG"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]', exitCode: '1', stdErr: '+ export 'CONTAINERD_SOCKET=/run/containerd/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID=ksniff-container-oZQpVNjG'
+ chroot /host ctr -a /run/containerd/containerd.sock task kill -s SIGKILL ksniff-container-oZQpVNjG
ctr: container "ksniff-container-oZQpVNjG" in namespace "k8s.io": not found
'  error="command terminated with exit code 1"
ERRO[0002] failed to remove privileged container: 'ksniff-privileged', exit code: '1', please manually remove it  error="command terminated with exit code 1"
INFO[0002] removing pod: 'ksniff-xvmg8'                 
INFO[0002] removing privileged pod: 'ksniff-xvmg8'      
INFO[0002] privileged pod: 'ksniff-xvmg8' removed       
INFO[0002] pod: 'ksniff-xvmg8' removed successfully     
INFO[0002] sniffer cleanup completed successfully       
Error: signal: killed

cmdjulian avatar Aug 26 '21 14:08 cmdjulian

Hello!

Can you please try running ksniff with the following argument? --socket /run/k3s/containerd/containerd.sock

bostrt avatar Aug 27 '21 20:08 bostrt

Yep, that worked for me executing it as sudo. Otherwise wireshark complains about not having proper permissions, see:

Couldn't run /usr/bin/dumpcap in child process: Keine Berechtigung

I think you can close the issue, thanks for your help! 😄

cmdjulian avatar Aug 27 '21 20:08 cmdjulian

same issue and I ran kubectl sniff prometheus-rancher-monitoring-prometheus-0 -n cattle-monitoring-system --privileged --socket /run/k3s/containerd/containerd.sock. but I got the following error

    set -ex
    export CONTAINERD_SOCKET="/run/k3s/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 5d967cee29d23f44eb24cfd916fb42e02b8a0e98ec865712386221ede3ef30db | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-sQzWcRMb tcpdump -i any -U -w -
    ]' on container: 'ksniff-privileged', pod: 'ksniff-hvnqf', namespace: 'cattle-monitoring-system'
ERRO[0006] failed executing command: '[/bin/sh -c
    set -ex
    export CONTAINERD_SOCKET=""
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID=""
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]', exitCode: '3', stdErr: '+ export 'CONTAINERD_SOCKET='
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID='
+ chroot /host ctr -a task kill -s SIGKILL
No help topic for 'kill'
'  error="command terminated with exit code 3"
ERRO[0006] failed to remove privileged container: 'ksniff-privileged', exit code: '3', please manually remove it  error="command terminated with exit code 3"
INFO[0006] removing pod: 'ksniff-hvnqf'
INFO[0006] removing privileged pod: 'ksniff-hvnqf'
INFO[0006] privileged pod: 'ksniff-hvnqf' removed
INFO[0006] pod: 'ksniff-hvnqf' removed successfully
INFO[0006] sniffer cleanup completed successfully

houshym avatar Aug 28 '21 01:08 houshym

@cmdjulian any clue? I tried that one and it does not work.

houshym avatar Aug 31 '21 01:08 houshym

The command which made it work for me was with the following:

sudo kubectl sniff some-deployment -p --socket /run/k3s/containerd/containerd.sock

I'm also using a quiet recent version of k3s, 1.21. Maybe thats an issue on your side as well?

cmdjulian avatar Sep 10 '21 19:09 cmdjulian

Unfortunately I cannot run it.

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4+k3s1", GitCommit:"3e250fdbab72d88f7e6aae57446023a0567ffc97", GitTreeState:"clean", BuildDate:"2021-08-19T19:09:53Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

INFO[0000] waiting for pod successful startup INFO[0002] pod: 'ksniff-8mgh9' created successfully on node: 'k8s-worker2' INFO[0002] spawning wireshark! INFO[0002] starting sniffer cleanup INFO[0002] removing privileged container: 'ksniff-privileged' INFO[0002] executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_ID="" chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID} ]' on container: 'ksniff-privileged', pod: 'ksniff-8mgh9', namespace: 'default' INFO[0002] starting remote sniffing using privileged pod INFO[0002] executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="/run/k3s/containerd/containerd.sock" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}" export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT} crictl pull docker.io/maintained/tcpdump:latest >/dev/null netns=$(crictl inspect d984d3da598c2c45e4eda4deb12c4ded21f5d1571d2fec87e187f0698f7250b4 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"') exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-VNBjddQy tcpdump -i any -U -w - ]' on container: 'ksniff-privileged', pod: 'ksniff-8mgh9', namespace: 'default' ERRO[0002] failed executing command: '[/bin/sh -c set -ex export CONTAINERD_SOCKET="" export CONTAINERD_NAMESPACE="k8s.io" export CONTAINER_ID="" chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID} ]', exitCode: '3', stdErr: '+ export 'CONTAINERD_SOCKET='

  • export 'CONTAINERD_NAMESPACE=k8s.io'
  • export 'CONTAINER_ID='
  • chroot /host ctr -a task kill -s SIGKILL No help topic for 'kill' ' error="command terminated with exit code 3" ERRO[0002] failed to remove privileged container: 'ksniff-privileged', exit code: '3', please manually remove it error="command terminated with exit code 3" INFO[0002] removing pod: 'ksniff-8mgh9' INFO[0002] removing privileged pod: 'ksniff-8mgh9' INFO[0002] privileged pod: 'ksniff-8mgh9' removed INFO[0002] pod: 'ksniff-8mgh9' removed successfully INFO[0002] sniffer cleanup completed successfully Error: exec: "wireshark": executable file not found in $PATH

houshym avatar Sep 10 '21 20:09 houshym