containerd
Supports multiple IP address assignment for `nerdctl run` and `nerdctl compose up`
Per (https://github.com/containerd/nerdctl/issues/3556#issuecomment-2466527814) I am opening an issue on what seems to be CNI concurrency issue.
I am having CNI issues when using multiple networks. One is macvlan network and the other is a bridge network. I've also attempted dual macvlan with similar results. What happens looks like an ordering issue. Sometimes the bridge interface would try and come up as a macvlan for example or as below shows the macvlan tries to come up as a bridge . I made a basic test compose file with hello-world and it happens there are well. Nothing is currently assigned the ip. I've tried other unused ip's as well.
I found this when trying to use 2.0.0-rc3 and I retried when 2.0.0 was released and the problem persists. The same compose file works without issue in docker compose.
This is the result from a "nerdctl compose up". The home.local is macvlan, proxy.home.local is a bridge. Single network services start with no issues.
# nerdctl compose up
INFO[0000] Ensuring image hello-world
INFO[0000] Creating container test
FATA[0003] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2024-11-09T11:06:52-06:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): failed to allocate all requested IPs: 10.0.0.171": unknown
FATA[0003] error while creating container test: exit status 1
The hello-world compose file:
services:
hello:
image: hello-world
container_name: test
hostname: test
networks:
home.local:
ipv4_address: 10.0.0.171
mac_address: 02:42:0a:00:01:47
proxy.home.local:
ipv4_address: 10.100.100.127
networks:
home.local:
name: home.local
external: true
proxy.home.local:
name: proxy.home.local
external: true
Network creation commands and cni versions:
# nerdctl network create -d macvlan --subnet=10.0.0.0/24 --gateway=10.0.0.1 -o parent=bond0 -o macvlan_mode=bridge home.local
# nerdctl network create --subnet=10.100.100.0/24 proxy.home.local
# /opt/cni/bin/macvlan -v
CNI macvlan plugin v1.6.0
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
# /opt/cni/bin/bridge -v
CNI bridge plugin v1.6.0
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
OK, here's the detail
I'm not sure if this a bug or not. For now, we do not support multiple IP address assignment yet.
https://github.com/containerd/nerdctl/blob/main/pkg/cmd/container/create.go#L634-L662
Your compose file will be converted to the command nerdctl run --pull=never --net=home.local --ip=10.0.0.171 --mac-address=02:42:0a:00:01:47 --net=proxy.home.local --ip=10.100.100.127 --hostname=test --restart=no hello-world
The last IP address will override the first IP address. So the issue comes here.
I'm not sure we should support multiple IP address assignment. WDYT @AkihiroSuda
I'm not sure we should support multiple IP address assignment.
It should correspond to whether Docker supports multiple IP address assignment
I'm not sure we should support multiple IP address assignment.
It should correspond to whether Docker supports multiple IP address assignment
Same idea, Let us figure it out
I have figured it out.
I have the same issue using the nerdctl that comes with Rancher Desktop on macOS. Switching to Docker fixes it. So, Docker supports multiple IP address assignment.
Docker does not support passing multiple networks and multiple ips on the run command: the last specified ip will be used for the first specified network (<- the logic in that escapes me, but whatever). I personally think this ^ is just buggy and should support multiple ips.
eg:
docker run -ti --net one --ip "10.42.0.42" --net two --ip "10.42.0.43" debian
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.42.0.43 netmask 255.255.255.0 broadcast 10.42.0.255
ether d6:a5:f6:5c:73:51 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.43.0.2 netmask 255.255.255.0 broadcast 10.43.0.255
ether 26:0a:28:de:1b:1c txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The same is likely true for mac and ipv6 as well. Same for port bindings.
I suspect what docker does with that compose file is create, then connect (for each network), then run.
If I am not mistaken, fixing this in nerdctl requires a good many different things to happen and likely significant effort:
- [ ] implement
network connectthat we apparently do not have - [ ] replace all labels for ip, ipv6, mac, to support slices instead of string (this will likely break backward compatibility if not done very carefully)
- [ ] rewrite a good section of ocihook (and maybe netutil)
- [ ] possibly rewrite compose logic to leverage
connectwhen presented with multiple networks
I need some of this.
Not personally interested in the compose part, but I need multi-ip with multi-networks, so, I am likely to go for 2. and 3, and either 1. or allow multi-ips to be specified on run.
If anyone else is interested in this and wants to chime in, give me a ping so that we either collaborate on this or at least don't duplicate effort.
Actually, docker DOES support specifying details for multiple networks on run - just with this syntax:
docker run -itd --network=name=my-net1,ip=192.0.2.42 --network=name=my-net2,ip=192.0.3.42 busybox