for-linux Docker does not handle a VRF setup

[x] This is a bug report
[ ] This is a feature request
[x] I searched existing issues before opening this one

Expected behavior

dockerd should be able to bind to IPs in VRFs. Or at least handle packets which are originating in a VRF correctly when net.ipv4.tcp_l3mdev_accept = 1 is enabled.

Actual behavior

dockerd is not able to bind to those IPs. It just aborts with: listen tcp <ip_in_vrf>:25: bind: cannot assign requested address

And even with net.ipv4.tcp_l3mdev_accept = 1 set it does not answer to packets from a VRF when listening to wildcard.

Steps to reproduce the behavior

Add a vrf to Linux and try to bind to an IP.

Output of docker version:

Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:36:00 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 11
 Running: 11
 Paused: 0
 Stopped: 0
Images: 11
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-9-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 3.863GiB
Name: mail
ID: VYBI:QWFI:ZJUC:STNW:BP26:63YV:I4YP:5IKW:4EDW:CNB5:OXCO:JICW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

It's running on Debian 9 in a KVM machine.

I hope it is understandable what I am trying to achieve. If you need more details I gather them :).

Jun 17 '19 08:06 awlx

I investigated a bit further, so it seems the iptables rules generated are only using the table "local" to get the list of local IPs: 324 28112 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL And all VRF IPs are obviously not in there.

ip route show table local type local
local 10.80.248.22 dev vlan1000 proto kernel scope host src 10.80.248.22 
local 10.80.255.31 dev lo proto kernel scope host src 10.80.255.31 
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 
local 172.17.0.1 dev docker0 proto kernel scope host src 172.17.0.1 
local 192.168.203.1 dev br-6f05b26bb02e proto kernel scope host src 192.168.203.1

So that's why those rules don't match.

Jun 17 '19 17:06 awlx

/cc @arkodg @euanh ptal

Jul 05 '19 08:07 thaJeztah

hi @awlx I haven't played around much with VRFs on Linux before, would appreciate it if you could share the exact steps performed to

create a VRF
create a docker container so I can reproduce the issue by running the commands as a script, thanks !

Jul 10 '19 21:07 arkodg

Hi,

Did anyone find a way to get around this issue?

Regards, Rajen

Mar 13 '20 00:03 rajendra-dendukuri

@arkodg Simple reproduction script here:

sysctl net.ipv4.tcp_l3mdev_accept=1
sysctl net.ipv4.udp_l3mdev_accept=1
modprobe dummy
ip link add vrf1 type vrf table 200
ip route add vrf vrf1 unreachable default metric 4278198272
ip link add dummy10 type dummy
ip link set dummy10 vrf vrf1
ip addr add 169.254.1.1/32 dev dummy10
docker run -itd --rm -p 169.254.1.1:80:80 debian:10-slim

Which will result in:

docker: Error response from daemon: driver failed programming external connectivity on endpoint vigilant_fermi (e8275973f7dca1ea42c91f414e5064d9
c6bf5ef103f4cd92e244cdba13909c11): Error starting userland proxy: listen tcp 169.254.1.1:80: bind: cannot assign requested address.

My docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:56 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683