gvisor
gvisor copied to clipboard
google
Running docker in gvisor quickstart not working with --bridge=none
Description
Hi, I was following along with the docker in gvisor quickstart guide. But when I try and build the whalesay container it doesn't work.
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
link/loopback 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 127.0.0.1/8 scope global dynamic
inet6 ::1/128 scope global dynamic
2: eth0: <UP,LOWER_UP> mtu 8980
link/ether 1a:fe:a1:37:66:a4 brd ff:ff:ff:ff:ff:ff
inet 172.29.88.163/32 scope global dynamic
inet6 fe80::18fe:a1ff:fe37:66a4/64 scope global dynamic
3: docker0: <UP,LOWER_UP> mtu 1486
link/ether 72:d4:1c:2e:9d:b9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global dynamic
/ # mkdir whalesay && cd whalesay
/whalesay # cat > Dockerfile <<EOF
> FROM ubuntu
>
> RUN apt-get update && apt-get install -y cowsay curl
> RUN mkdir -p /usr/share/cowsay/cows/
> RUN curl -o /usr/share/cowsay/cows/docker.cow https://raw.githubusercontent.com/docker/whalesay/master/docker.cow
> ENTRYPOINT ["/usr/games/cowsay", "-f", "docker.cow"]
> EOF
/whalesay #
/whalesay # docker build -t whalesay .
[+] Building 8.2s (5/7) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 309B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 1.6s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/4] FROM docker.io/library/ubuntu:latest@sha256:440dcf6a5640b2ae5c77724e68787a906afb8ddee98bf86db94eea8528c2c076 3.4s
=> => resolve docker.io/library/ubuntu:latest@sha256:440dcf6a5640b2ae5c77724e68787a906afb8ddee98bf86db94eea8528c2c076 0.0s
=> => sha256:440dcf6a5640b2ae5c77724e68787a906afb8ddee98bf86db94eea8528c2c076 6.69kB / 6.69kB 0.0s
=> => sha256:dbdff34bb41cecdb07c79af373b44bb4c9ccba2520f014221fb95845f14bc6c1 424B / 424B 0.0s
=> => sha256:f9248aac10f2f82e0970222e36cc7b71215b88e974e001282e5cd89797a82218 2.30kB / 2.30kB 0.0s
=> => sha256:b08e2ff4391ef70ca747960a731d1f21a75febbd86edc403cd1514a099615808 29.72MB / 29.72MB 0.7s
=> => extracting sha256:b08e2ff4391ef70ca747960a731d1f21a75febbd86edc403cd1514a099615808 2.4s
=> ERROR [2/4] RUN apt-get update && apt-get install -y cowsay curl 3.1s
------
> [2/4] RUN apt-get update && apt-get install -y cowsay curl:
3.108 runc run failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: failed to add interface vethe67ea93 to sandbox: failed to subscribe to link updates: permission denied
3.108
------
Dockerfile:3
--------------------
1 | FROM ubuntu
2 |
3 | >>> RUN apt-get update && apt-get install -y cowsay curl
4 | RUN mkdir -p /usr/share/cowsay/cows/
5 | RUN curl -o /usr/share/cowsay/cows/docker.cow https://raw.githubusercontent.com/docker/whalesay/master/docker.cow
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y cowsay curl" did not complete successfully: exit code: 1
/whalesay #
--net-rawis enabled- I am using the following pod definition:
apiVersion: v1
kind: Pod
metadata:
name: docker-node011
spec:
runtimeClassName: gvisor
nodeName: node011
containers:
- name: docker
image: docker:dind
securityContext:
privileged: true
capabilities:
add: ["ALL"]
command:
- /bin/sh
- -c
- |
set -xe -o pipefail
# Grab the first default route interface
dev=$(ip route show default | awk '/default/ {print $5; exit}')
echo "Default interface: $dev"
# Get the IPv4 address associated with that interface
addr=$(ip -4 addr show dev "$dev" | awk '/inet / {print $2}' | cut -d/ -f1)
echo "Interface address: $addr"
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables-legacy -t nat -A POSTROUTING -o "$dev" -j SNAT --to-source "$addr" -p tcp
iptables-legacy -t nat -A POSTROUTING -o "$dev" -j SNAT --to-source "$addr" -p udp
exec dockerd --iptables=false --ip6tables=false -D
volumeMounts:
- name: docker
mountPath: /var/lib/docker
volumes:
- name: docker
emptyDir: {}
restartPolicy: Never
EOF
Steps to reproduce
Create a pod with gvisor using the following pod definition:
apiVersion: v1
kind: Pod
metadata:
name: docker-node011
spec:
runtimeClassName: gvisor
nodeName: node011
containers:
- name: docker
image: docker:dind
securityContext:
privileged: true
capabilities:
add: ["ALL"]
command:
- /bin/sh
- -c
- |
set -xe -o pipefail
# Grab the first default route interface
dev=$(ip route show default | awk '/default/ {print $5; exit}')
echo "Default interface: $dev"
# Get the IPv4 address associated with that interface
addr=$(ip -4 addr show dev "$dev" | awk '/inet / {print $2}' | cut -d/ -f1)
echo "Interface address: $addr"
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables-legacy -t nat -A POSTROUTING -o "$dev" -j SNAT --to-source "$addr" -p tcp
iptables-legacy -t nat -A POSTROUTING -o "$dev" -j SNAT --to-source "$addr" -p udp
exec dockerd --iptables=false --ip6tables=false -D
volumeMounts:
- name: docker
mountPath: /var/lib/docker
volumes:
- name: docker
emptyDir: {}
restartPolicy: Never
EOF
And try and build the whalesay container:
/ # mkdir whalesay && cd whalesay
/whalesay # cat > Dockerfile <<EOF
> FROM ubuntu
>
> RUN apt-get update && apt-get install -y cowsay curl
> RUN mkdir -p /usr/share/cowsay/cows/
> RUN curl -o /usr/share/cowsay/cows/docker.cow https://raw.githubusercontent.com/docker/whalesay/master/docker.cow
> ENTRYPOINT ["/usr/games/cowsay", "-f", "docker.cow"]
> EOF
/whalesay #
/whalesay # docker build -t whalesay .
[+] Building 8.2s (5/7) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 309B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 1.6s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/4] FROM docker.io/library/ubuntu:latest@sha256:440dcf6a5640b2ae5c77724e68787a906afb8ddee98bf86db94eea8528c2c076 3.4s
=> => resolve docker.io/library/ubuntu:latest@sha256:440dcf6a5640b2ae5c77724e68787a906afb8ddee98bf86db94eea8528c2c076 0.0s
=> => sha256:440dcf6a5640b2ae5c77724e68787a906afb8ddee98bf86db94eea8528c2c076 6.69kB / 6.69kB 0.0s
=> => sha256:dbdff34bb41cecdb07c79af373b44bb4c9ccba2520f014221fb95845f14bc6c1 424B / 424B 0.0s
=> => sha256:f9248aac10f2f82e0970222e36cc7b71215b88e974e001282e5cd89797a82218 2.30kB / 2.30kB 0.0s
=> => sha256:b08e2ff4391ef70ca747960a731d1f21a75febbd86edc403cd1514a099615808 29.72MB / 29.72MB 0.7s
=> => extracting sha256:b08e2ff4391ef70ca747960a731d1f21a75febbd86edc403cd1514a099615808 2.4s
=> ERROR [2/4] RUN apt-get update && apt-get install -y cowsay curl 3.1s
------
> [2/4] RUN apt-get update && apt-get install -y cowsay curl:
3.108 runc run failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: failed to add interface vethe67ea93 to sandbox: failed to subscribe to link updates: permission denied
3.108
------
Dockerfile:3
--------------------
1 | FROM ubuntu
2 |
3 | >>> RUN apt-get update && apt-get install -y cowsay curl
4 | RUN mkdir -p /usr/share/cowsay/cows/
5 | RUN curl -o /usr/share/cowsay/cows/docker.cow https://raw.githubusercontent.com/docker/whalesay/master/docker.cow
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y cowsay curl" did not complete successfully: exit code: 1
/whalesay #
runsc version
root@node011:/var/log/runsc/203d7ede0ecb087ae67e46421b4824a873a0419890acdef92a80223a25a0c0a2# runsc --version
runsc version release-20250625.0
spec: 1.2.0
docker version (if using docker)
uname
Linux node011 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
kubectl (if using Kubernetes)
root@node011:/var/log/runsc/203d7ede0ecb087ae67e46421b4824a873a0419890acdef92a80223a25a0c0a2# kubectl version
Client Version: v1.31.5
Kustomize Version: v5.4.2
Server Version: v1.31.5
repo state (if built from source)
No response
runsc debug logs (if available)
I have enabled debug logging but there are loads of logs, happy to add them if people have better ideas of what is needed/useful for resolution
@gmintoco, thank you for the report.
This error is triggered by https://github.com/moby/moby/commit/eaa84bc8f43739a9f813ea7e72feb2e6135caf7c. It was introduced in Docker v28.0, so any version prior to it should work without this issue. We're working on supporting new Docker versions.
@avagin is correct.
a question is that you start the dockerd with bridge driver from the yaml file you provided at exec dockerd --iptables=false --ip6tables=false -D, I don't see where you specify --bridge=none
Thanks a lot for your response! Let me try out a previous version as well :)
@avagin I tried with old version but still didnt work.
i am testing with GKE-1.33.5
apiVersion: v1
kind: Pod
metadata:
name: simple-dind-gvisor
labels:
app: dind-gvisor-test
spec:
runtimeClassName: gvisor
restartPolicy: Never
volumes:
- name: docker-socket
emptyDir: {}
- name: docker-storage
emptyDir: {}
containers:
# -----------------------------
# Docker daemon (dockerd)
# -----------------------------
- name: docker-daemon
image: docker:27.5.1-dind
args:
- dockerd
- --storage-driver=vfs
- --host=unix:///var/run/docker.sock
- --iptables=false
- --ip-masq=false
- --bridge=none
securityContext:
privileged: false
capabilities:
add:
- AUDIT_WRITE
- CHOWN
- DAC_OVERRIDE
- FOWNER
- FSETID
- KILL
- MKNOD
- NET_BIND_SERVICE
- NET_RAW
- SETFCAP
- SETGID
- SETPCAP
- SETUID
- SYS_CHROOT
- SYS_PTRACE
- NET_ADMIN
- SYS_ADMIN
volumeMounts:
- name: docker-socket
mountPath: /var/run
- name: docker-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"
# -----------------------------
# Docker client (tests)
# -----------------------------
- name: docker-client
image: docker:27.5.1-cli
env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
command: ["sh", "-c"]
args:
- |
echo "Waiting for Docker daemon..."
until docker info >/dev/null 2>&1; do
sleep 2
done
echo "=== Docker version ==="
docker version
echo "=== Docker daemon ==="
docker info
echo "=== Docker run test ==="
docker run --rm hello-world || true
echo "=== Docker build test ==="
cat <<EOF > Dockerfile
FROM alpine
RUN echo "Hello from Docker inside gVisor"
EOF
docker build -t gvisor-test .
docker run --rm gvisor-test
echo "=== Done. Sleeping ==="
sleep 3600
volumeMounts:
- name: docker-socket
mountPath: /var/run
resources:
requests:
cpu: "100m"
memory: "128Mi"
Error is below
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [1/2] FROM [docker.io/library/alpine:latest@sha256:51183f2cfa6320055da30872f211093f9ff1d3cf06f39a0bdb212314c5dc7375](http://docker.io/library/alpine:latest@sha256:51183f2cfa6320055da30872f211093f9ff1d3cf06f39a0bdb212314c5dc7375)
#4 resolve [docker.io/library/alpine:latest@sha256:51183f2cfa6320055da30872f211093f9ff1d3cf06f39a0bdb212314c5dc7375](http://docker.io/library/alpine:latest@sha256:51183f2cfa6320055da30872f211093f9ff1d3cf06f39a0bdb212314c5dc7375) done
#4 sha256:a107a3c031732299dd9dd607bb13787834db2de38cfa13f1993b7105e4814c60 1.02kB / 1.02kB done
#4 sha256:7acffee03fe864cd6b88219a1028855d6c912e7cf6fac633aa4307529fd0cc08 611B / 611B done
#4 sha256:014e56e613968f73cce0858124ca5fbc601d7888099969a4eea69f31dcd71a53 0B / 3.86MB 0.1s
#4 sha256:51183f2cfa6320055da30872f211093f9ff1d3cf06f39a0bdb212314c5dc7375 9.22kB / 9.22kB done
#4 sha256:014e56e613968f73cce0858124ca5fbc601d7888099969a4eea69f31dcd71a53 3.86MB / 3.86MB 0.1s done
#4 extracting sha256:014e56e613968f73cce0858124ca5fbc601d7888099969a4eea69f31dcd71a53 0.1s done
#4 DONE 0.2s
#5 [2/2] RUN echo "Hello from Docker inside gVisor"
#5 ERROR: process "/bin/sh -c echo \"Hello from Docker inside gVisor\"" did not complete successfully: network bridge not found
------
> [2/2] RUN echo "Hello from Docker inside gVisor":
------
Dockerfile:2
--------------------
1 | FROM alpine
2 | >>> RUN echo "Hello from Docker inside gVisor"
3 |
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c echo \"Hello from Docker inside gVisor\"" did not complete successfully: network bridge not found
Unable to find image 'gvisor-test:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Run 'docker run --help' for more information
I reproduce the issue, and i think gvsior is working as intended for your case @imranrazakhan
the problem is that you are trying to build a image gvisor-test in gvisor sandbox, however, your dockerd has no bridge network, which by default dockerd will use for your docker build command, your build command attempts pull alpine image from internet based on your Dockerfile
$ kubectl exec -it simple-dind-gvisor -c docker-client -- docker network ls
NETWORK ID NAME DRIVER SCOPE
8073d8a4999b host host local
4557a043848d none null local
when you enable bridge=none you don't expect any internet via bridge device (the default option for docker client) unless you specify host network like
kubectl exec -it simple-dind-gvisor -c docker-client -- sh
/test # cat <<EOF > Dockerfile
> FROM alpine:latest
> CMD ["echo", "hello"]
> EOF
/test # docker build --network host -t gvisor-test .
/test # docker run gvisor-test
hello
@milantracy Thanks for updates, i want to use dind+gvisor for github runner, so what should be value of bridge and what will be the impact of using --network host , is it secure?