for-linux icon indicating copy to clipboard operation
for-linux copied to clipboard
verified publisher icon docker

Rootless docker on Ubuntu 18.04 Kernel 4.9 is not working with internet, iptable errors

Open KCSesh opened this issue 4 years ago • 5 comments

  • [x] This is a bug report
  • [ ] This is a feature request
  • [x] I searched existing issues before opening this one

Expected behavior

Rootless docker to work with slirp4netns or vpnkit on Ubuntu 18.04 Kernel 4.9.

Actual behavior

Iptable error and if I disable it, my container does not have internet access but my username space does.

failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.2: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)
[rootlesskit:child ] error: command [./dockerd-rootless.sh ] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1

Steps to reproduce the behavior

1. ./dockerd-rootless.sh

Causes the IP table error.

I can successfully start it by doing:

1. ./dockerd-rootless.sh --iptables=false

But then internet in my container does not work. But if I nsenter -U --preserve-credentials -n -m -t $(cat /run/user/1000/docker.pid)

I can curl within the namespace. This error happens with both vpnkit and slirp4netns.

I am looking for a path forward. Even if this means manually setting up the iptable rules with root. Where can I find the iptable commands that need to be run?

I am also looking for help in determining why I have access in the user namespace but not the container?

Output of docker version:

Docker version 20.10.2, build 20.10.2-0ubuntu1~18.04.2

Output of docker info:


Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 4.9.140
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 31.05GiB
 Name: tegra-ubuntu
 ID: XG27:723O:37YX:D5JK:MKLP:F6PT:43U2:BMCW:KLLZ:GW5U:N7GW:A42G
 Docker Root Dir: /home/<user>/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled


**Additional environment details (AWS, VirtualBox, physical, etc.)**
Physical

I also can't get it to work in nerdctl or containerd as rootless. But podman rootless does work.

https://github.com/containerd/nerdctl/issues/190#issuecomment-828597192

KCSesh avatar May 08 '21 01:05 KCSesh

Please try sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter

AkihiroSuda avatar May 10 '21 18:05 AkihiroSuda 

user1@tegra-ubuntu:/data/bin$ sudo modprobe --verbose ip_tables iptable_mangle iptable_nat iptable_filter
user1@tegra-ubuntu:/data/bin$ ./dockerd-rootless.sh
+ [ -w /run/user/1000 ]
+ [ -w /home/user1/ ]
+ rootlesskit=
+ which docker-rootlesskit
+ which rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ :
+ :
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z ]
+ which slirp4netns
+ slirp4netns --help
+ grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ [ -z ]
+ _DOCKERD_ROOTLESS_CHILD=1
+ export _DOCKERD_ROOTLESS_CHILD
+ id -u
+ [ 1000 = 0 ]
+ exec rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave ./dockerd-rootless.sh
+ [ -w /run/user/1000 ]
+ [ -w /home/user1/ ]
+ rootlesskit=
+ which docker-rootlesskit
+ which rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ :
+ :
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z ]
+ which slirp4netns
+ slirp4netns+  --help
grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ [ -z 1 ]
+ [ 1 = 1 ]
+ rm -f /run/docker /run/containerd /run/xtables.lock
+ exec dockerd
INFO[2021-05-10T18:35:24.148876420Z] Starting up
WARN[2021-05-10T18:35:24.149095342Z] Running in rootless mode. This mode has feature limitations.
INFO[2021-05-10T18:35:24.149168434Z] Running with RootlessKit integration
INFO[2021-05-10T18:35:24.157284838Z] libcontainerd: started new containerd process  pid=20735
INFO[2021-05-10T18:35:24.157827103Z] parsed scheme: "unix"                         module=grpc
INFO[2021-05-10T18:35:24.157954341Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-05-10T18:35:24.158080522Z] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-05-10T18:35:24.158133133Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-05-10T18:35:24.229148197Z] starting containerd                           revision= version="1.4.4-0ubuntu1~18.04.2"
INFO[2021-05-10T18:35:24.300353830Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2021-05-10T18:35:24.300739896Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.307563601Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/4.9.140\\n\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.307759194Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.309396741Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /home/user1/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.309503530Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2021-05-10T18:35:24.309658353Z] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2021-05-10T18:35:24.309718196Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.309989888Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.310327760Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.311458627Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /home/user1/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-05-10T18:35:24.311573033Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2021-05-10T18:35:24.311663181Z] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2021-05-10T18:35:24.311721167Z] metadata content store policy set             policy=shared
INFO[2021-05-10T18:35:24.312239879Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2021-05-10T18:35:24.312324747Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2021-05-10T18:35:24.312489139Z] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.312722813Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.312868004Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.312993354Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.313075534Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.313159505Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.313241877Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.313325721Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.313391804Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2021-05-10T18:35:24.313726507Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
ERRO[2021-05-10T18:35:24.314257700Z] cleanup working directory in namespace        error="open /home/user1/.local/share/docker/containerd/daemon/io.containerd.runtime.v2.task/moby: no such file or directory" namespace=moby
INFO[2021-05-10T18:35:24.314372169Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2021-05-10T18:35:24.316691539Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2021-05-10T18:35:24.316878844Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2021-05-10T18:35:24.317188202Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317280366Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317367218Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317441174Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317508281Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317610910Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317698530Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317771717Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.317910123Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2021-05-10T18:35:24.318207545Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.318334047Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.318443492Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.318509095Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2021-05-10T18:35:24.319194854Z] serving...                                    address=/run/user/1000/docker/containerd/containerd-debug.sock
INFO[2021-05-10T18:35:24.319489044Z] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock.ttrpc
INFO[2021-05-10T18:35:24.319705374Z] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock
INFO[2021-05-10T18:35:24.319790433Z] containerd successfully booted in 0.093526s
INFO[2021-05-10T18:35:24.345675524Z] parsed scheme: "unix"                         module=grpc
INFO[2021-05-10T18:35:24.345807018Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-05-10T18:35:24.346002259Z] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-05-10T18:35:24.346074295Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-05-10T18:35:24.348554952Z] parsed scheme: "unix"                         module=grpc
INFO[2021-05-10T18:35:24.348645581Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-05-10T18:35:24.348727664Z] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-05-10T18:35:24.348776179Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-05-10T18:35:24.387886132Z] [graphdriver] using prior storage driver: overlay2
WARN[2021-05-10T18:35:24.399036019Z] Your kernel does not support CPU realtime scheduler
WARN[2021-05-10T18:35:24.399099798Z] Your kernel does not support cgroup blkio weight
WARN[2021-05-10T18:35:24.399130904Z] Your kernel does not support cgroup blkio weight_device
INFO[2021-05-10T18:35:24.399674001Z] Loading containers: start.
WARN[2021-05-10T18:35:24.415881400Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.6.2: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3
INFO[2021-05-10T18:35:24.556772372Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2021-05-10T18:35:24.558419424Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2021-05-10T18:35:24.558589864Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.2: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)
[rootlesskit:child ] error: command [./dockerd-rootless.sh] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1

Same error unfortunately.

KCSesh avatar May 10 '21 18:05 KCSesh

Is there any update on this? I'm seeing the same issue running on debian:buster-slim

veleek avatar Oct 18 '21 22:10 veleek

I'm having this issue in Debian 10 as well. I'm not sure but I think it's because slirp4netns is v0.2 something installed from apt in Debian 10 and the Docker docs suggest v.0.4. But it's a catch-22 because according to "install from source" dependencies for slirp4netns:

https://github.com/rootless-containers/slirp4netns#install-from-source

it says libslirp-dev is required and that package is not available in apt for Debian 10.

antimomentum avatar Dec 22 '22 13:12 antimomentum

I'm having this issue in Debian 10 as well. I'm not sure but I think it's because slirp4netns is v0.2 something installed from apt in Debian 10 and the Docker docs suggest v.0.4. But it's a catch-22 because according to "install from source" dependencies for slirp4netns:

https://github.com/rootless-containers/slirp4netns#install-from-source

it says libslirp-dev is required and that package is not available in apt for Debian 10.

libslirp can be installed from the source too https://gitlab.freedesktop.org/slirp/libslirp

Anyway, Debian 10 has reached EOL and is not recommended

AkihiroSuda avatar Dec 22 '22 14:12 AkihiroSuda