docker icon indicating copy to clipboard operation
docker copied to clipboard

Published 20 hours ago • verified publisher icon openwrt

docker interface eth0 has no ip when run with --privilege flag

Open keshavmass0 opened this issue 3 years ago • 9 comments

When I execute docker run command, it has below impacts Without privileged flag -- ($ docker run -itd openwrtorg/rootfs:x86-64) Internet is accessible on container Ip was assigned to docker eth0 interface With privileged flag -- ($ docker run -itd --privileged openwrtorg/rootfs:x86-64) Internet not accessible on container no IP attached to docker eth0 interface, but IP is visible in inspect command output

With --privileged flag we have root privileges but in our case IP is not attached to eth0 primary docker interface. I just wanna confirm why this strange behavior is being observed.

Thanks keshav

keshavmass0 avatar Feb 08 '21 16:02 keshavmass0

I actually wound up investigating this very same issue today and found myself here. I see the same issue when running a rootfs container with either --privileged or --cap-add=NET_ADMIN. After tearing my hair out for quite a while, I discovered that eth0 does in fact have an address initially, it just get removed once the container has finished starting up:

# run with default entrypoint
➜  ~ CONTAINER=$(docker run --rm -it -d --privileged openwrtorg/rootfs:x86-64-openwrt-21.02)
# address is initially configured
➜  ~ docker exec $CONTAINER ip addr list dev eth0
147: eth0@if148: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd00::242:ac11:2/80 scope global flags 02 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever
# sleep for a little bit to allow init process to finish
➜  ~ sleep 5
# address on eth0 is gone
➜  ~ docker exec $CONTAINER ip addr list dev eth0
149: eth0@if150: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever

Using the --entrypoint option, we can override the default entrypoint (/sbin/init) and set it to /bin/ash. This stops the address from getting removed:

➜  ~ CONTAINER=$(docker run --rm -it -d --privileged --entrypoint /bin/ash openwrtorg/rootfs:x86-64-openwrt-21.02)
➜  ~ docker exec $CONTAINER ip addr list dev eth0
147: eth0@if148: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd00::242:ac11:2/80 scope global flags 02 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever

Therefore I assume that by giving the container extra privileges, something that runs during the normal init process ends up wiping the address from eth0. When not running with --privileged or --cap-add=NET_ADMIN, perhaps the address remains because the container doesn't have permission to remove it?

Is there a simple way to prevent the address on eth0 from being wiped during the init process? For my own purposes, I want to use the normal init process since I want all the normal services to be running within the container, thus simply overriding it with --entrypoint doesn't help me. Any ideas?

nwidger avatar Jul 09 '21 19:07 nwidger

Yep, good find :detective: The issue is that OpenWrt tries to takeover control of the interface and apply it's default configuration (dhcp) rather than using the setup interface provided by Docker. This happens pretty much here.

Running ash directly rather than the whole /sbin/init stuff prevents this behaviour, however the running container also lacks a running ubusd and uhttpd instance, aka most services wouldn't work nor the LuCI web interface.

Question is for what do we want to use the OpenWrt containers. I'm happy to change them so that only ash is executed and users start whatever they like manually.

aparcar avatar Jul 09 '21 21:07 aparcar

@aparcar I don't know where it comes from or what other (possibly negative) consequences this might have, but setting network.lan.device in /tmp/board.json to an empty string seems to stop the address from being flushed. Is this a really bad idea? If no, would it be possible to build the roots images with this change?

Personally, I like that /sbin/init is run and that all the services one would expect are brought up. I'd think making the user need to know exactly which services should be started would be a bit of a pain, as that would be dependent on what packages are installed and how they are configured, presumably.

nwidger avatar Jul 09 '21 23:07 nwidger

All right, i that case we should make the init system "container aware" like we did some time ago with procd. I'll think about it but feel free to join in

aparcar avatar Jul 14 '21 07:07 aparcar

@aparcar Is there a simpler way (perhaps by overriding some files via https://github.com/openwrt/docker/tree/master/rootfs) to just tell OpenWrt to not touch eth0? The rootfs images already contain slight modifications so that they run better in Docker, the current situation is just unfortunate in that giving the contain more privileges actually makes it function worse.

nwidger avatar Jul 14 '21 16:07 nwidger

@nwidger I figured out a way to tape that preinit issue by adding

RUN echo $'\n\
preinit_ip() {\n\
        echo "- disabling preinit_ip -"\n\
}\n' >> /lib/preinit/11_disable_preinit_network

@aparcar I found an old discussion where you suggested a similar solution. but is there a better way without disabling network service? so that I would like to manage other interface other than eth0. surprisingly other interfaces(eg. eth3, eth4) aren't getting flushed by starting network service.

my network config and netifd logs showing undesired manipulation of eth0 without explicit configuration.

> /etc/config/network
config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
 
 
config interface 'lan'
        option device 'eth2'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
 
 
config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'
 
> logread with netifd debug enabled
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: device_create_default(622): Create simple device 'eth0'
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: device_init_virtual(576): Initialize device 'eth0'
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: cb_clear_event(1053): Remove a route from device eth0
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: cb_clear_event(1053): Remove an address from device eth0
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: device_set_present(757): Network device 'eth0' is now present
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: __device_add_user(835): Add user for device 'eth0', refcount=1
Thu Sep 23 20:32:35 2021 daemon.err netifd[2235]: device_claim(510): Claim Network device eth0, new active count: 1

xavier83 avatar Sep 23 '21 19:09 xavier83

@aparcar I don't know where it comes from or what other (possibly negative) consequences this might have, but setting network.lan.device in /tmp/board.json to an empty string seems to stop the address from being flushed. Is this a really bad idea? If no, would it be possible to build the roots images with this change?

Personally, I like that /sbin/init is run and that all the services one would expect are brought up. I'd think making the user need to know exactly which services should be started would be a bit of a pain, as that would be dependent on what packages are installed and how they are configured, presumably.

Aw that is cool find. @nwidger led me to this (partial) solution. RUN sed -i 's/eth0/eth2/g' /etc/board.d/99-default_network does the trick for x86_64. but not so much on armvirt-64 where something else again seems to flush eth0 after config_init_wireless which isn't happening on x86_64 @aparcar any idea on what it might be?

netifd on x86_64:

Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: proto_shell_add_handler(939): Add handler for script ./ppp.sh: pppoe
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: interface_update(1469): Create interface 'loopback'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_create_default(622): Create simple device 'lo'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_init_virtual(576): Initialize device 'lo'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1053): Remove a route from device lo
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1053): Remove a route from device lo
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1053): Remove an address from device lo
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_set_present(757): Network device 'lo' is now present
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: __device_add_user(835): Add user for device 'lo', refcount=1
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: interface_set_available(479): Interface 'loopback', available=1
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: __device_add_user(835): Add user for device 'lo', refcount=2
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: interface_update(1469): Create interface 'lan'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_create_default(622): Create simple device 'eth2'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_init_virtual(576): Initialize device 'eth2'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: __device_add_user(835): Add user for device 'eth2', refcount=1
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: __device_add_user(835): Add user for device 'eth2', refcount=2
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: interface_update(1469): Create interface 'wan'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_create_default(622): Create simple device 'eth1'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_init_virtual(576): Initialize device 'eth1'
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: __device_add_user(835): Add user for device 'eth1', refcount=1
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: __device_add_user(835): Add user for device 'eth1', refcount=2
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1049): Remove a rule
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1049): Remove a rule
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1049): Remove a rule
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1049): Remove a rule
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: cb_clear_event(1049): Remove a rule
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: config_init_wireless(607): No wireless configuration found
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: device_claim(510): Claim Network device lo, new active count: 1
Fri Sep 24 08:10:58 2021 daemon.notice netifd: Interface 'loopback' is enabled
Fri Sep 24 08:10:58 2021 daemon.notice netifd: Interface 'loopback' is setting up now
Fri Sep 24 08:10:58 2021 daemon.err netifd[2331]: interface_queue_event(123): Queue hotplug handler for interface 'loopback', event 'ifup'

netifd log on armvirt64

Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: proto_shell_add_handler(939): Add handler for script ./ppp.sh: pppoe
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: interface_update(1469): Create interface 'loopback'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_create_default(622): Create simple device 'lo'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_init_virtual(576): Initialize device 'lo'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1053): Remove a route from device lo
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1053): Remove a route from device lo
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1053): Remove an address from device lo
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_set_present(757): Network device 'lo' is now present
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'lo', refcount=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: interface_set_available(479): Interface 'loopback', available=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'lo', refcount=2
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: interface_update(1469): Create interface 'lan'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_create_default(622): Create simple device 'eth2'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_init_virtual(576): Initialize device 'eth2'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_set_present(757): Network device 'eth2' is now present
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'eth2', refcount=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: interface_set_available(479): Interface 'lan', available=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'eth2', refcount=2
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: interface_update(1469): Create interface 'wan'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_create_default(622): Create simple device 'eth1'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_init_virtual(576): Initialize device 'eth1'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_set_present(757): Network device 'eth1' is now present
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'eth1', refcount=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: interface_set_available(479): Interface 'wan', available=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'eth1', refcount=2
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1049): Remove a rule
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1049): Remove a rule
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1049): Remove a rule
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1049): Remove a rule
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1049): Remove a rule
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: config_init_wireless(607): No wireless configuration found
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_claim(510): Claim Network device eth2, new active count: 1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_create_default(622): Create simple device 'eth0'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_init_virtual(576): Initialize device 'eth0'
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1053): Remove a route from device eth0
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: cb_clear_event(1053): Remove an address from device eth0
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_set_present(757): Network device 'eth0' is now present
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: __device_add_user(835): Add user for device 'eth0', refcount=1
Fri Sep 24 07:31:24 2021 daemon.err netifd[1493]: device_claim(510): Claim Network device eth0, new active count: 1
Fri Sep 24 07:31:24 2021 daemon.notice netifd: Interface 'lan' is enabled

Edit: it seems to disable the eth0 flushing if there is only one interface even on arm64. maybe I'm missing something :thinking:

Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: proto_shell_add_handler(939): Add handler for script ./ppp.sh: pppoe
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: interface_update(1469): Create interface 'loopback'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_create_default(622): Create simple device 'lo'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_init_virtual(576): Initialize device 'lo'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1053): Remove a route from device lo
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1053): Remove a route from device lo
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1053): Remove an address from device lo
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_set_present(757): Network device 'lo' is now present
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: __device_add_user(835): Add user for device 'lo', refcount=1
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: interface_set_available(479): Interface 'loopback', available=1
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: __device_add_user(835): Add user for device 'lo', refcount=2
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: interface_update(1469): Create interface 'lan'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_create_default(622): Create simple device 'eth2'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_init_virtual(576): Initialize device 'eth2'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: __device_add_user(835): Add user for device 'eth2', refcount=1
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: __device_add_user(835): Add user for device 'eth2', refcount=2
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: interface_update(1469): Create interface 'wan'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_create_default(622): Create simple device 'eth1'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_init_virtual(576): Initialize device 'eth1'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: __device_add_user(835): Add user for device 'eth1', refcount=1
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: __device_add_user(835): Add user for device 'eth1', refcount=2
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1049): Remove a rule
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1049): Remove a rule
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1049): Remove a rule
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1049): Remove a rule
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: cb_clear_event(1049): Remove a rule
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: config_init_wireless(607): No wireless configuration found
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: device_claim(510): Claim Network device lo, new active count: 1
Fri Sep 24 11:53:02 2021 daemon.notice netifd: Interface 'loopback' is enabled
Fri Sep 24 11:53:02 2021 daemon.notice netifd: Interface 'loopback' is setting up now
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: interface_queue_event(123): Queue hotplug handler for interface 'loopback', event 'ifup'
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: call_hotplug(99): Call hotplug handler for interface 'loopback', event 'ifup' (lo)
Fri Sep 24 11:53:02 2021 daemon.notice netifd: Interface 'loopback' is now up
Fri Sep 24 11:53:02 2021 daemon.notice netifd: Network device 'lo' link is up
Fri Sep 24 11:53:02 2021 daemon.notice netifd: Interface 'loopback' has link connectivity
Fri Sep 24 11:53:02 2021 daemon.err netifd[2140]: interface_queue_event(123): Queue hotplug handler for interface 'loopback', event 'iflink'
Fri Sep 24 11:53:03 2021 daemon.err netifd[2140]: task_complete(107): Complete hotplug handler for interface 'loopback'

xavier83 avatar Sep 24 '21 06:09 xavier83

It is looking like as long as there is some network in /etc/config/network mentions interface that actually exists in the host, netifd auto-magically flushes eth0 as well when it sets up those configured networks.

Sigh :disappointed: it seems like a dead-end on both sides https://github.com/k8snetworkplumbingwg/multus-cni/issues/256 :worried:

xavier83 avatar Sep 25 '21 14:09 xavier83

I have compiled the rootfs for OpenWrt myself and it works properly. Basically, I use the Mavlan network for Podman.

daiaji avatar Feb 01 '24 16:02 daiaji