ksniff
ksniff copied to clipboard
eldadru
netns return empty string so nothing is ever captured
Hi, I'm really interested into testing ksniff. I just install it on an AWS standard AMI with a local microk8s cluster installed along with docker as container runtime.
I have a simple webserver where I launch ksniff on. But there is never a single packet coming in wireshark (or stdout when I launch just ksniff with -o -). Of course I make in parralel plenty of HTTP request to the simple webserver "bgd"
After issuing a ps command, I can see how the ksniff container command has been run and I thinks this may be the cause of my problem but I don't know how to specify a net (like net=host ? but will it work ?)
# Command was
kubectl sniff -n default bgd-xxx-yyy
INFO[0005] executing command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
crictl pull docker.io/maintained/tcpdump:latest >/dev/null
netns=$(crictl inspect 5a0038d78596179fa055aa1395ae515f13011e06751b2271131a07f5a87ba27f | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-sluEcVnE tcpdump -i any -U -w -
]' on container: 'ksniff-privileged', pod: 'ksniff-t4r9t', namespace: 'default'
# ps gives
4613 root 0:00 ctr -a /run/containerd/containerd.sock run --rm --with-ns network: docker.io/maintained/tcpdump:latest ksniff-container-sluEcVnE tcpdump -i any -U -w -
4670 root 0:00 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ksniff-container-sluEcVnE -address /run/containerd/containerd.sock
Notice the --with-ns network:
Any help is welcome and I can of course give more details, please ask
The "app" I'm using for this test (image: quay.io/redhatworkshops/bgd:latest) and yaml is :
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: bgd
name: bgd
spec:
replicas: 1
selector:
matchLabels:
app: bgd
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: bgd
spec:
containers:
- image: quay.io/redhatworkshops/bgd:latest
name: bgd
env:
- name: COLOR
value: "green"
resources: {}
Still searching from my side, I just see that despite I'm installing docker runtime through yum on the AMI, it also installs containerd. SO microk8s seems to use containerd and this is why ksniff uses the containerd config. Mays this be linked with my problem ?
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
docker x86_64 20.10.17-1.amzn2 amzn2extra-docker 39 M
Installing for dependencies:
containerd x86_64 1.6.6-1.amzn2 amzn2extra-docker 27 M
pigz x86_64 2.3.4-1.amzn2.0.1 amzn2-core 81 k
runc x86_64 1.1.3-1.amzn2 amzn2extra-docker 2.9 M
I'm really not sure of what container runtime microk8s is using... https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/#:~:text=If%20the%20%2D%2Dcontainer%2Druntime%2Dendpoint%20flag%20is%20present%2C,sock%20is%20the%20containerd%20endpoint.
sudo tr \\0 ' ' < /proc/"$(pgrep kubelet)"/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.14.268-205.500.amzn2.x86_64 root=UUID=f22063bb-28ab-42bc-ba62-b9130b861ee7 ro console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 LANG=en_US.UTF-8 KEYTABLE=us
According to the link: if the --container-runtime flag is not remote, you use the dockershim socket with Docker Engine
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.3-2+63243a96d1c393", GitCommit:"63243a96d1c3932c704cf0625f5834411bc3b2e6", GitTreeState:"clean", BuildDate:"2022-07-19T17:57:14Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
I just try to switch microk8s to docker for container-runtime (https://stackoverflow.com/questions/61119975/is-it-possible-to-set-microk8s-to-use-docker-engine-instead-of-containerd ) but it doesn't work, the cluster do not launch any pods and no events
Kubelet logs are clearer after checking (sorry)
août 26 08:13:51 ip-172-31-16-109.ec2.internal microk8s.daemon-kubelite[3913]: Error: failed to validate kubelet flags: unsupported CRI runtime: "docker", only "remote" is curre
For information, kubelet conf is as following
sudo cat /var/snap/microk8s/current/args/kubelet
--kubeconfig=${SNAP_DATA}/credentials/kubelet.config
--cert-dir=${SNAP_DATA}/certs
--client-ca-file=${SNAP_DATA}/certs/ca.crt
--anonymous-auth=false
--root-dir=${SNAP_COMMON}/var/lib/kubelet
--fail-swap-on=false
--feature-gates=DevicePlugins=true
--eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi"
--container-runtime=remote
--container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
--containerd=${SNAP_COMMON}/run/containerd.sock
--node-labels="microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane"
--authentication-token-webhook=true
--read-only-port=0
--cluster-domain=cluster.local
--cluster-dns=10.152.183.10
With help of ps, I guess there is a mismatch between containerd socket used by microk8s and the one (default containerd) where ksniff is inspecting the containers to get network infos
ps -aux |grep containerd
root 12852 0.0 0.8 1445188 36012 ? Ssl 08:02 0:00 /usr/bin/containerd
root 12898 0.0 1.6 1361004 64944 ? Ssl 08:02 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536
root 19418 6.6 1.1 1912688 47836 ? Ssl 08:21 0:20 /snap/microk8s/3597/bin/containerd --config /var/snap/microk8s/3597/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock
root 20604 0.0 0.2 710504 8940 ? Sl 08:21 0:00 /var/lib/snapd/snap/microk8s/3597/bin/containerd-shim-runc-v1 -namespace k8s.io -id 2f4b67f8277aa44692ef96c691fd9bfd3186f90b91b28557c9614fec0173f659 -address /var/snap/microk8s/common/run/containerd.sock
root 21063 0.0 0.2 710504 8448 ? Sl 08:21 0:00 /var/lib/snapd/snap/microk8s/3597/bin/containerd-shim-runc-v1 -namespace k8s.io -id e02a1a73114c7c32cb302d91d0fba8f11c83a3d4dbf912e329d8078f2b4a1d8f -address /var/snap/microk8s/common/run/containerd.sock
...
--address /var/snap/microk8s/common/run/containerd.sock
When I use the ctr command from microk8s, I can have kind of inspect but the jq command is not working. The problem is : how to give ksniff the info about the contaierd sock to use : seems related to : https://github.com/eldadru/ksniff/issues/65
sudo /snap/bin/microk8s.ctr container info fbc451b82bdfbef2fd91c3fa3625448ee6690228dec033cde56f7c4b2920776f
sudo /snap/bin/microk8s.ctr container info fbc451b82bdfbef2fd91c3fa3625448ee6690228dec033cde56f7c4b2920776f | jq .info
null
Grrr, I don't understand
kubectl sniff -n default bgd-7b56ddcf4b-jnvzb -p --socket /var/snap/microk8s/common/run/containerd.sock
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'bgd'
INFO[0000] sniffing method: privileged pod
INFO[0000] sniffing on pod: 'bgd-7b56ddcf4b-jnvzb' [namespace: 'default', container: 'bgd', filter: '', interface: 'any']
INFO[0000] creating privileged pod on node: 'ip-172-31-16-109.ec2.internal'
INFO[0000] pod: 'ksniff-jc7pp' created successfully in namespace: 'default'
INFO[0000] waiting for pod successful startup
INFO[0005] pod: 'ksniff-jc7pp' created successfully on node: 'ip-172-31-16-109.ec2.internal'
INFO[0005] spawning wireshark!
INFO[0005] starting remote sniffing using privileged pod
INFO[0005] executing command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
crictl pull docker.io/maintained/tcpdump:latest >/dev/null
netns=$(crictl inspect 6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-YIVNUQiY tcpdump -i any -U -w -
]' on container: 'ksniff-privileged', pod: 'ksniff-jc7pp', namespace: 'default'
INFO[0005] starting sniffer cleanup
INFO[0005] removing privileged container: 'ksniff-privileged'
INFO[0005] executing command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_ID="ksniff-container-YIVNUQiY"
chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
]' on container: 'ksniff-privileged', pod: 'ksniff-jc7pp', namespace: 'default'
INFO[0005] command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_ID="ksniff-container-YIVNUQiY"
chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
]' executing successfully exitCode: '1', stdErr :'+ export 'CONTAINERD_SOCKET=/var/snap/microk8s/common/run/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID=ksniff-container-YIVNUQiY'
+ chroot /host ctr -a /var/snap/microk8s/common/run/containerd.sock task kill -s SIGKILL ksniff-container-YIVNUQiY
ctr: container "ksniff-container-YIVNUQiY" in namespace "k8s.io": not found
'
INFO[0005] privileged container: 'ksniff-privileged' removed successfully
INFO[0005] removing pod: 'ksniff-jc7pp'
INFO[0005] removing privileged pod: 'ksniff-jc7pp'
INFO[0005] privileged pod: 'ksniff-jc7pp' removed
INFO[0005] pod: 'ksniff-jc7pp' removed successfully
INFO[0005] sniffer cleanup completed successfully
Error: exit status 1
container "ksniff-container-YIVNUQiY" in namespace "k8s.io": not found
But...
sudo /snap/bin/microk8s.ctr namespaces ls
NAME LABELS
k8s.io
sudo /snap/bin/microk8s.ctr container info 6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9 | jq .Image
"quay.io/redhatworkshops/bgd:latest"
sudo /snap/bin/microk8s.ctr container ls | grep 6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9
6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9 quay.io/redhatworkshops/bgd:latest io.containerd.runc.v1
Hello, still searching if you have any ideas. I did the whole test https://github.com/eldadru/ksniff/issues/65#issuecomment-984790977 reffered by @skyhisi including wrapper for /usr/bin/ctr (I made a backup of the ctr binary)
Thi works a bit better (wireshark is lauching and the right network seems to be found) but an another error occur :
ctr: failed to create temp dir: stat /run/user/0/snap.microk8s: no such file or directory
The whole log is :
kubectl sniff -n default bgd-7b56ddcf4b-hpxr7 -p --socket /var/snap/microk8s/common/run/containerd.sock
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'bgd'
INFO[0000] sniffing method: privileged pod
INFO[0000] sniffing on pod: 'bgd-7b56ddcf4b-hpxr7' [namespace: 'default', container: 'bgd', filter: '', interface: 'any']
INFO[0000] creating privileged pod on node: 'ip-172-31-20-62.ec2.internal'
INFO[0000] pod: 'ksniff-8sbzx' created successfully in namespace: 'default'
INFO[0000] waiting for pod successful startup
INFO[0003] pod: 'ksniff-8sbzx' created successfully on node: 'ip-172-31-20-62.ec2.internal'
INFO[0003] spawning wireshark!
INFO[0003] starting remote sniffing using privileged pod
INFO[0003] executing command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
crictl pull docker.io/maintained/tcpdump:latest >/dev/null
netns=$(crictl inspect a5023573788d6a67593ad80567c486337646e100db8d071d70535c17774f58ec | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-RIvqkpPo tcpdump -i any -U -w -
]' on container: 'ksniff-privileged', pod: 'ksniff-8sbzx', namespace: 'default'
INFO[0004] command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
crictl pull docker.io/maintained/tcpdump:latest >/dev/null
netns=$(crictl inspect a5023573788d6a67593ad80567c486337646e100db8d071d70535c17774f58ec | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-RIvqkpPo tcpdump -i any -U -w -
]' executing successfully exitCode: '1', stdErr :'+ export 'CONTAINERD_SOCKET=/var/snap/microk8s/common/run/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_RUNTIME_ENDPOINT=unix:///host/var/snap/microk8s/common/run/containerd.sock'
+ export 'IMAGE_SERVICE_ENDPOINT=unix:///host/var/snap/microk8s/common/run/containerd.sock'
+ crictl pull docker.io/maintained/tcpdump:latest
+ tr -d '"'
+ jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path'
+ crictl inspect a5023573788d6a67593ad80567c486337646e100db8d071d70535c17774f58ec
+ netns=/proc/1591/ns/net
+ exec chroot /host ctr -a /var/snap/microk8s/common/run/containerd.sock run --rm --with-ns network:/proc/1591/ns/net docker.io/maintained/tcpdump:latest ksniff-container-RIvqkpPo tcpdump -i any -U -w -
ctr: failed to create temp dir: stat /run/user/0/snap.microk8s: no such file or directory
'
### Here it hangs while wireshark is launched (trying desperatly to show some capture packets that never comes
### Following lines appears when I close wireshark
INFO[0004] remote sniffing using privileged pod completed
INFO[0022] starting sniffer cleanup
INFO[0022] removing privileged container: 'ksniff-privileged'
INFO[0022] executing command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_ID="ksniff-container-RIvqkpPo"
chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
]' on container: 'ksniff-privileged', pod: 'ksniff-8sbzx', namespace: 'default'
INFO[0022] command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_ID="ksniff-container-RIvqkpPo"
chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
]' executing successfully exitCode: '1', stdErr :'+ export 'CONTAINERD_SOCKET=/var/snap/microk8s/common/run/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID=ksniff-container-RIvqkpPo'
+ chroot /host ctr -a /var/snap/microk8s/common/run/containerd.sock task kill -s SIGKILL ksniff-container-RIvqkpPo
ctr: container "ksniff-container-RIvqkpPo" in namespace "k8s.io": not found
'
INFO[0022] privileged container: 'ksniff-privileged' removed successfully
INFO[0022] removing pod: 'ksniff-8sbzx'
INFO[0022] removing privileged pod: 'ksniff-8sbzx'
INFO[0022] privileged pod: 'ksniff-8sbzx' removed
INFO[0022] pod: 'ksniff-8sbzx' removed successfully
INFO[0022] sniffer cleanup completed successfully
hey @seb54000 i just want to acknowledge your issue. I haven't dug into yet but I plan to soon. The microk8s bridge for ksniff is pretty complicated so there's a lot to look at, as you have!
Thanks @bostrt , no hurry here, it is just for testing. When we will figure out how to cope with my microk8s config I will be happy to contribute a README in the repo for this "specific" use case.