VXLAN can only send message but does not pick up the incoming message from host

[sonkanit]

I have setup 2 docker swarm nodes on 2 VMs, both of them are manager. However the first VM, called NAT VM, is using NAT interface with proper port forwarding rules. The second is using bridge interface and the swarm is created by this node. docker swarm init --advertise-addr

The swam join was successful on NAT VM. docker swarm join --advertise-addr --listen-addr <local NAT ip>

All nodes are reachable and active as seen by "docker node ls" The test services are running on both nodes.

The problem occurs when I tried to ping from the container running in NAT node to the container on the other end.

Here is the result from tcpdump -i enps03 port 4789 on NAT VM. Noticed the "echo replies" were received. 16:20:16.415449 IP nat_node.58620 > 192.168.1.108.4789: VXLAN, flags [I] (0x08), vni 4097 IP 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 4746, length 64 16:20:16.417093 IP _gateway.43645 > nat_node.4789: VXLAN, flags [I] (0x08), vni 4097 IP 10.0.0.3 > 10.0.0.6: ICMP echo reply, id 72, seq 4746, length 64 16:20:17.417299 IP nat_node.58620 > 192.168.1.108.4789: VXLAN, flags [I] (0x08), vni 4097

However when I enter the network namespace and do tcpdump, I only see the "echo requests" but not the "echo replies" 16:26:27.043329 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5116, length 64 16:26:28.045229 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5117, length 64 16:26:29.047107 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5118, length 64

I think there could be something wrong with my configuration. I am not sure how to do further investigation on this issue.

Thank you.

Client: Debug Mode: false

Server: Containers: 2 Running: 2 Paused: 0 Stopped: 0 Images: 3 Server Version: 19.03.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries s Swarm: active NodeID: m0kkg71w5p3z25vg090g380j1 Is Manager: true ClusterID: w1g7axnt3sv9mniimcooh51h1 Managers: 2 Nodes: 2 Default Address Pool: 10.0.0.0/8 SubnetSize: 24 Data Path Port: 4789 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 10 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 192.168.1.109 Manager Addresses: 192.168.1.108:2377 192.168.1.109:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-58-generic Operating System: Ubuntu 18.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 985.5MiB Name: nat_node ID: GHF5:I7RQ:UX4Z:Q3NM:SKO7:XJUX:H2XW:O6IB:KHSG:PKBM:NCGX:WPAR Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false